Questions For This Week

What would a test run of the program look like outside USASOC?

Who among our existing contacts can help us with another kind of test run?

What does USASOC need from us to have a more complete timeline on deployment?

To what extent will we need to be involved with USASOC once we reach the deployment stage?

What We Are Testing

• Our program generates enough interest from DoD entities to spur action.
• Our program is functional enough to manage our correspondence.
• Our program has a clear path to business viability.
• We understand what data we need to pull from USASOC’s databases in our deployment.

Key Insights

From a USASOC deployment standpoint, we were able to confirm that many users are as interested in AARs and SITREPs as they are in things like trip reports and RFOs. We also developed a better characterization of what these forms are providing. There is still no formal process for identifying a problem at the outset, trip reports cover available options, SITREPs cover a project’s timeline, and AARs cover the end results of field testing and deployment. We also informed our problem sponsor that a testable prototype is imminent and are awaiting his response as to what next steps will be.

From a wider DoD standpoint we are still exploring the potential to deploy with a unit like AFWERX following our test run with USASOC. However, our conversation with Mike at DIU offered us some insight as to what our endgame could be if we focus on deploying within DoD. According to Mike, DIU currently deconflicts projects with 70 entities across the department and largely does so manually with phone calls and meeting. He acknowledged the challenge of trying to deploy this kind of solution at such a scale but was surprisingly optimistic about it. He felt that if we could get individual buy in from these entities, then the technical challenges associated with connecting them would not be so difficult to overcome.

From a business viability standpoint, Dr. Jared expressed some skepticism as to whether this program can differentiate itself enough from what is out there to avoid getting crushed by established competition. To succeed as a business, we will either need to be able to find that clear differentiating factor that delivers clear value or find a customer niche large enough to sustain our business but small enough to avoid attention from the largest competitors. Our discussion with Akber also indicated that our particular method of deployment, while it could be a differentiating factor, is also likely to be more expensive from a deployment standpoint than a cloud solution, particularly if our aim is to expand to the commercial sector.

Interviews (10/122)

Akber – Fire Engine Red

• His system is cloud based, so if we are deploying on site it may be more costly.
• Sales team can generally manage 6 clients per person at a given time.
• Deploying a product with a client typically takes about 3 months and the equivalent of 1.5 monthly salaries between sales and tech.
• Tech also manages troubleshooting and maintenance, maintenance is much lower cost than rollout.

Brian – Squadron F

• From a collaboration standpoint, he’ll do so with his team but also with 5-6 other offices whose projects might have technical or operational overlap.
• In particular, he wants to be able to easily collaborate even when the other team might physically be halfway around the world. For an operation it can be for weeks/months, for a longer term development it could be over a year.
• Right now that is accomplished with emails and in person meetings. Can be difficult to get everyone in the room at the same time due to packed daily schedules and frequent travel.

Alex – BMNT

• Expressed interest in our project.
• Believes that there are pathways to move it forward, either via MD5 or some other DoD program.
• He will go and see what could be available to us and circle around with next steps.

MSG Darrell – R&D Section Leader

• The way I want CDD involved in my projects depends on the project itself, which is why I would want to have some choice of how much access each user or shop has.
• In some cases I’m tied in from the beginning, which could include going on trips on their behalf and reporting back.
• In other situations, I will only tie them in when I need access to a specific funding line.

Dr. Jared – Course Advisor – BMNT

• Believes we identified “an interesting set of needs.”
• However, is also concerned about the competitive landscape as it pertains to the program’s business viability.
• To make a business out of it you’ll have to really connect with a customer OR identify a segment that’s profitable for you but too small for you to get crushed by a giant competitor.

Mike – DIU

• Project deconfliction is an increasing priority, as is collaborating across the ecosystems that populate DoD.
• There are 70 entities in the innovation space, all working to deconflict. Many have direct overlap with our portfolios. Weekly or even daily we will meet with them to deconflict and we regularly get questions from appropriators on how we will deconflict.
• We also like to shop successful prototypes around to give other agencies a chance to jump on.
• Some sort of automated tool to manage this external process would make life easier, however we are confident in our internal process and don’t feel it would be as helpful in that context.

Nic – Squadron R&D Deployment

• The best data points for our purposes are the application pieces coming from the end users in squadron, often in the form of sitreps and AARs. However, there is no uniform method to submit this kind of info.
• SITREPs are like living documents, designed to keep everyone up to date on projects.
• AARs are more like end documents/statements, saying we went into this with X intent, what we want to sustain, what we want to change.
• Other types of data that is useful: what environment the tool is applied in; consumer purchase methodology (easy to use? need training? size, weight, power).

Tim – NSTXL CEO

• In his experience, meeting the military where they are when it comes to workflows, processes, and cultures is the best way to move projects forward.
• That said, if they are buying in to your system then they’ll take some suggestion on how to use it.
• Offer some best practices with the program but don’t expect everyone to behave that way. Don’t bother with rules, they’re only going to follow your guidelines if they think it will be useful.

SGM Angel – CDD – Future Concepts

• The critical squadrons from our perspective are Signal, Intel, and H. That is where much of the emerging tech exploration is taking place.
• Also engage with F and G – they are nice to have and not irrelevant.
• However, you need to make sure when you are using terms that are not standardized in the military that everyone is talking about the same thing.

Dr. Jared – Course Advisor – BMNT

• Having a good idea for a program is just the start when we are talking transitioning to business. You have to execute it and find something that makes users want to adopt it.
• Often that something is a behavioral insight that isn’t core to the tech itself. The best executed tech doesn’t always have the best users.
• While better execution will help you in some places, if you want to grow your user base to a level that is sustainable as a business, it has to enable people to do something that they could not have done in that way before.

Hypotheses Confirmed/Debunked

• Partially Confirmed: Our program generates enough interest from DoD entities to spur action. This week offered further confirmation that other DoD entities recognize USASOC’s problem as relevant to other units and are interested in helping us build the program. However, we have not found any new units (other than the ones mentioned in previous blog posts) that expressed interest in a test deployment of our program. We are confident we can get people’s wheels turning, particularly in innovation spaces like AFWERX, but less confident in exactly how far that will take us.
• Partially Confirmed: Our program is functional enough to manage our correspondence. To test our program’s functionality, we loaded all of our beneficiary interviews onto the platform to make sure we could easily add them and search them. While not fully functional, this transition was largely successful. However, it was a small number of files and applied by the person who created the program, so it’s less clear whether that ease of use will translate to the end user experience.
• Partially Debunked: Our program has a clear path to business viability. Our interviews with Dr. Jared made it clear that even though our program could be highly effective, we will likely need to find a clear differentiating factor that separates our program from all the others in the field to be a sustainable business model. As such, we don’t see this week’s interviews as totally dismissive of our potential, but rather that we will likely need to take more concrete steps to demonstrate value to the end user. Dr. Jared’s suggestion of finding a small niche isn’t totally unrealistic either – if we succeed in deploying at USASOC where other software failed that could be indicative of our ability to make this kind of program work within a government space.
• Confirmed: We understand what data we need to pull from USASOC’s databases in our deployment. It seemed almost every week we heard about some new potential source of data from which we could draw. But this week’s interviews with our direct beneficiaries showed that we seem to have a handle on the key types of information that end users care about. We were also able to clarify the timeline our end users have in mind when trying to access different types of forms.

Questions For Next Week

Now that the prototype is ready, when can we start testing?

What level of access will we have to deployment data?

Can we adapt our program to another unit’s workflow or is it too specific to USASOC?

What is our most viable path for further deployment of the program? Should we pursue niche government innovation spaces or the business sector?

 

Questions For This Week

What should our combined interface include?

What is the range of opinions on what the division of labor between CDD and R&D should be?

What would a test run of the program look like outside USASOC?

Who among our existing contacts can help us with another kind of test run?

What We Are Testing

• Our input template can cover USASOC’s range of input types.
• Our combined interface is navigable and easy to understand without much prompting.
• Organizations outside of USASOC would want to test this program.

Key Insights

Regarding division of labor, Dave from Squadron H offered us our most valuable insight. He noted that end users real pain point is having to keep pushing a problem in the unit when they have things in the field to worry about. As such, they’re going to reach out to whoever they think will be most responsive. For any particular division of responsibilities to work, it will need to include a timeline that involves communication with the end user at certain key points in the process. R&D section leader Josh suggested that CDD should at minimum always be in the room and ideally always in charge of talking to vendors. The big problem with other people doing it is they don’t have the training to make sure they aren’t accidentally committing the unit to a purchase that they’re not actually committed to making.

Regarding our interface, our most valuable insight was to minimize the size of the blank spaces that go with our prompts. If we deliberately ask for concise answers to questions, we can simultaneously cut down on fluff and be less intimidating to the folks who aren’t big on typing.

Regarding outside opportunities, our most valuable insight came from Chris at BMNT. He suggested that the kind of problem we are working on comes through H4D and BMNT’s doors all the time and that he could even think of a project they’re working on where it may be viable to test our program. This is particularly significant given the range of DoD and federal entities that BMNT works with; if our program turns out to be a viable solution for the kinds of problems they are talking about then it would be a significant validation of the value we bring.

We are continuing the discussion with our problem sponsor about what deployment might look like but there have not been any significant updates to that process since last week.

Interviews (10/112)

Nic – Squadron R&D Deployment

• Would prefer pre-RWS submissions be submitted via an emailed link.
• Sees it as no different from submitting a visitor request, which people already do all the time.
• Difference is a pre-RWS submission wouldn’t be mandatory so people would need to be motivated to do it.

Mike – CAG in support of JSOC [Almost Ready MVP Feedback]

• Regarding filling out forms, likes the idea of light templates with prompts but thinks they should all be optional.
• Emphasis should be on concise responses, maybe even phrasing each template as a question could help with that.
• Ideally there will be a handful of pre-built templates to manage all the different forms, but also the ability to create a new template.

MAJ Josh – R&D Section Leader

• Believes CDD should be involved in potential procurements as soon as possible.
• Ideally, nobody talks to a vendor except CDD or at least CDD has reps in the room during the discussion.
• CDD personnel receive special procurement training to avoid talking themselves into an unauthorized commitment by the government. Your average Joe in the unit has a higher chance of getting themselves into trouble.
• CDD also has more of a birds eye view of what the organization needs – they are receiving requirements from everyone so they can help synchronize efforts and reduce duplication better than on our side.

MSG Joe – CDD – C4I

• Sees relationship between R&D and CDD as continuing to evolve.
• Believes he has a solid understanding of how things should work in his areas.
• Not certain about the “ideal” division of responsibilities.

Dr. Jared – Course Advisor – BMNT

• Deploying software without it being maintained isn’t really a viable solution unless you can get people to volunteer to look after it.
• One of the best reasons to start a company is to make sure that people actually use this thing and not let it languish.
• The only other option is giving the customer full ownership.

MSG Dave – H/A Squadrons

• The big problem regarding division of labor is that people get into their own little bubbles and have the people they contact who they think will get the job done, myself included.
• There isn’t exactly a consistent designation of roles, so when we do find a unique solution to our problem elsewhere in the unit it’s an extremely lucky break.
• A big pain for end users is that they don’t have the time to constantly push a capability gap – they want to send it to whoever will get the job done and get back to other things.
• People tend to get caught up in their own projects, which can divert the timelines on important capabilities.

Chris – BMNT

• Excited to hear about our project – many of the problems that come through BMNT’s door have to do with knowledge management.
• There may be an opportunity to test the program with a real problem they are working on.
• Similar situation profile: relying on email is causing problems. People don’t know who to send things to, there are limited number of mailing lists, and people are afraid to talk about niche things because of the spam factor.

Brian – Signal Squadron

• Doesn’t necessarily think he should be involved in every meeting, nor does he necessarily have the time.
• That said, he would like to know about stakeholder/face to face meetings as early as possible so he can adjust his schedule if he is interested enough.
• People have no problem injecting themselves into the situation when they find it relevant, but they need the time to find out and adjust accordingly.

Mike – CAG in support of JSOC [updated MVP]

• Emphasized limiting the amount of blank space on the forms.
• Feels this will be a psychological advantage for folks who are averse to writing.
• It would be nice to be able to search the answers in the prompts in addition to the tags.
• Thinks the focus needs to be twofold at this stage: bringing down the “cost of participation” for trip goers (i.e. time taken to fill things out), and making sure the input is useful for analysis and available somewhere in HQ.

Archie – NSTXL

• Current job is to build up his capability to network different H4D teams together so we can share best practices.
• He is meeting with a team called “Lumineye” soon that could be of interest to us.
• H4D’s most prominent success story is a company called Capella Space. However, it should be noted that they were already a startup before the course and primarily used it to further their existing goals.
• Wants to connect us to post-H4D capabilities as they get built.

Hypotheses Confirmed/Debunked

• Confirmed: Our input template can cover USASOC’s range of input types. Our current set up appears to be sufficient to gather relevant information on a variety of activities and could even ask for more concise answers, assuming that we can provide a few different versions of the template for the most standard inputs. This may create its own interface challenge as we need to make sure a range of input templates would not become overwhelming. However, it appears we are on the right track in terms of looks and how much info we are asking for.
• Partially Confirmed: Our combined interface is navigable and easy to understand without much prompting. Thus far, everyone we have talked to about the latest MVP seems to understand the functions. However, they are also people we’ve been talking to about this project all semester, so it remains unclear whether someone who is unfamiliar with the program would be able to jump on and figure things out right away. We will need to find a few people who are totally unfamiliar with what we are doing to really confirm this hypothesis.
• Confirmed: Organizations outside of USASOC would want to test this program. Chris from BMNT suggested that they work on knowledge management problems all the time and he could think of a current project where our solution might be applicable. No test plans with other organizations have actually been confirmed, but this is the strongest indication yet that we will have other opportunities to deploy the program. We are still in contact with Tony from AFWERX, but it seems the kinds of tests he had in mind involved us linking up with BMNT or another knowledge management firm so it is unclear whether AFWERX itself would want to help us test.

Questions for Next Week

What is USASOC’s perceived deployment timeline for this program?

Is our interface understandable to someone who hasn’t spent months in contact with us?

What would deploying on BMNT’s project look like? When would they expect it to happen?

What steps do we need to take to make sure we are prepared to manage relationships outside USASOC?

Questions for this Week

Can we fully confirm/debunk our hypotheses from last week that required more feedback?

For the forms that were confirmed useful – are we able to take the steps necessary to actually pull that data from its source?

What will our first deployment test look like?

Who will help us deploy the program?

What other opportunities exist to deploy this program in other units?

What We are Testing

• Beneficiaries see value in being able to see informal problem statements to which comments, updates, and trip reports could be attached. Our program can collect this data.
• Beneficiaries wish to share informal information sources such as news articles. Our program can collect this data.
• Beneficiaries want email notifications for Requests for Orders. Our program can collect this data.
• Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. Our program can collect this data.

Key Insights

From a product standpoint, much of this week was about ensuring the technical viability of the functions we’ve discussed up to this point and further confirming the value of collecting certain kinds of data with different beneficiaries. The trouble with collecting trip reports is well established and it seems AARs and SITREPs will require a similar system to collect, so we will know for sure if we are gathering those appropriately once we do a live test. O&M Logs and Travel Vouchers will be easy to collect and can be tied to system reminders to submit things like trip reports and AARs. RFOs are also easy to collect and the form could even be modified so that we can easily identify which trips apply to tech scouting.

From a deployment standpoint, we spoke with signal squadron about exactly how the process would work. They would first deploy it in a test environment with test users to understand the configurations and in the meantime set up a Change Advisory Board to determine the configuration for actual deployment. Due to the complexity of configurations on government systems, it’s unclear exactly how long such a process would take.

From a partnership standpoint, we spoke with a representative from Decisive Analytics, who saw the immediate value in our data gathering system working in tandem with their data processing system. We also opened up the prospect of speaking to other firms in the tech scouting world through our AFWERX contact. While such talks are preliminary, it appears that our concept is capable of stirring interest outside the context of our problem sponsor.

Interviews (10/102)

Brian – Signal Squadron

• Believes squadrons should be able to state or describe their problems in AARs.
• Sees value in CDD being able to look at all AARs and see how many people are complaining about the same capability gap. [noteworthy – he is the first person outside CDD to say this]
• Hopes that the program will enable a back and forth between groups that doesn’t require a ton of additional meetings or sifting through paperwork.

MAJ Tony – AFWERX

• Proposed connecting us with companies working in the tech scouting/tech adoption space that are not as sophisticated in the execution of software development.
• Acknowledged that security clearances could be a challenge. Did not think we could just get them on the possibility of the program being successful.
• Suggested that the best way forward is to make the MVP for unclassified activities. If our MVP is successful in an unclassified environment and could be useful for classified information, that can “prove” your need for clearance to take the project to the next level.

SGM Angel – CDD – Future Concepts

• Holds the general belief that squadrons should handle scouting for emerging tech that is currently available in industry.
• Was surprised to hear that some squadrons believed CDD should handle those kinds of activities. [worth noting that we did not use Angel’s distinction of tech that is currently available on the market when talking to squadrons]
• Thinks preferences may be personality dependent. But hopefully the unit reorganization will manage the difference in personalities – the Squadron rep and CDD rep will be side by side making decisions on who should go where under what circumstances.

Brian – Signal Squadron – Network Operations

• Trip reports are out there but not in one place that’s easy to find.
• RFO forms can be pulled. Will include dates, may be vague on trip details. We are trying to implement the use of a code to classify the kind of trip. Two categories in particular pertain to the tech scouting element. O&M log forms could also be pulled pretty easily.
• Travel Vouchers, AARs, and SITREPs are out there as well and we can get to them, but we kind of have to know what we’re looking for. Won’t just be able to pull from one place.
• We already have a database of vendors you can pull from as well.

MSG Dave – H/A Squadrons

• Does not believe CDD reorganization plan will solve the unit’s problems.
• Current job is to locate with A squadron to report their needs back to H squadron, and even though he’s there it’s a challenge to get in tune with what they are doing.
• Suspects that problem would repeat itself between CDD and the specialty squadrons.

MSG Joe – CDD – C4I

• Thinks challenges associated with RFO could be fixed with a vendor field and knows they already have the database.
• Sees our program, the Decisive Analytics program, and a new project management database for CDD as “the trifecta.”
• Eventually, you might be able to pull some of the data you need from that PM database.

Josh – Amazon R&D

• Works upstream from development teams.
• Amazon is highly decentralized – this comes with some inefficiencies but it could be more costly to take too much time thinking about a centraliezd policy.
• That said, his role is to risk assess a wide range of projects, then send the appropriately derisked ones down to R&D.
• Believes he is generally up to speed with the teams but the effort is active and informal. Holds tech “speed dating” meetings twice a year.

SGM Katie – CDD – Tech Targeting

• RFO descriptions are generally vague and not particularly informative.
• Does not think this is on purpose and could probably be changed to an extent, but unsure they could affect all elements in the building.
• Inevitably there will be some blind spots on the description itself, but knowledge of the trip itself and some of the other details on the form can be valuable.

Mark – Decisive Analytics

• Already worked with several other DoD agencies, achieved necessary clearances.
• Liked the idea of bouncing some of the data we would collect, particularly trip reports, off web services.
• Suggested there were other ways of working together if this particular project did not work out.

Chris – Signal Squadron – Network Operations

• To deploy a piece of software like yours, we would have a team of engineers deploy it in a test environment first to figure out configurations.
• Then we can bring in users to test it, push it through the Change Advisory Board (implementation) process while that is happening.
• Government networks have a lot of configuration, CAB is just there to make sure it all happens functionally.
• Should be workable to allow access based on roles within unit and clearances.

Hypotheses Confirmed/Debunked

• Confirmed: Beneficiaries see value in being able to see informal problem statements to which comments, updates, and trip reports could be attached. Interviewees have consistently expressed interest in being able to see problem statements. Partially confirmed: Our program can collect this data. Beneficiaries suggested such data could be pulled from the After Action Report process, though these forms suffer from the same inconsistency challenges of trip reports.
• Partially Confirmed: Beneficiaries wish to share informal information sources such as news articles. Beneficiaries do not consistently express interest in this but some are highly interested. Partially Confirmed: Our program can collect this data. It could be challenging for our program to incorporate this kind of information by itself but Decisive Analytics seems to have a better handle on the elements that make this data challenging to collect. As such, it’s more within the realm of possibility.
• Debunked: Beneficiaries want email notifications for Requests for Orders. There are certain kinds of info on RFOs that could be valuable and help to make building out trip reports more easy. As such, beneficiaries see them as helpful within the system. However, they do not want to look at them directly as they’re not informative enough on their own. Confirmed: Our program can collect this data. Not only are RFOs readily available for collection, we could slightly alter the existing form to easily categorize trips relevant to our system and set up reminders to submit trip reports.
• Debunked: Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. Beneficiaries again identified some information on the forms that could be useful, but would not want to look them up directly. Confirmed: Our program can collect this data. The database itself is readily available. If the information on the forms were to be configured appropriately it could yield some helpful information. However, given the general lack of enthusiasm it is low priority.

Questions for Next Week

What would a test run of the program look like outside USASOC?

Who among our existing contacts can help us with another kind of test run?

What does USASOC need from us to have a more complete timeline on deployment?

To what extent will we need to be involved with USASOC once we reach the deployment stage?

Questions for this Week

Is our input system more convenient than whatever people are doing now?

What amount of limited information is even the most paranoid R&D employee willing to share freely?

How can the search interface be condensed effectively? What kind of info should a condensed version display?

What value can CDD deliver that makes communicating with them worth the risk of getting a project shut down?

What use cases do our beneficiaries find most valuable? How many forms could our input system replace? Which forms do beneficiaries see direct value in searching?

What We are Testing

• Beneficiaries see value in searching and subscribing to After Action Reports.
• Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor.
• Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
• Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it.
• Beneficiaries wish to share informal information sources such as news articles.
• Beneficiaries want email notifications for relevant trip reports.
• Beneficiaries want email notifications for Requests for Orders.
• Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems.
• R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD.

Key Insights

This week torpedoed some of our assumptions regarding how much information we could glean from certain forms and how much value our beneficiaries saw in accessing them. But it also torpedoed some of our assumptions about how much people would be willing to participate in our system. Request for Orders forms, O&M log forms, and even requirements all met some skepticism when we described their case uses on our system. In each case, we will need to determine if we can consistently pull specific types of information from those forms because we can’t rely on people digging deeper the way we would with trip reports.

However, we were also surprised to see the level of interest in having the ability to submit informal pre-requirement problem statements. We were also relieved to find out that R&D’s concerns about CDD interference would not need to be completely eliminated in order to submit their data. Indeed, our program can even be framed as shielding R&D from certain kinds of unwanted contacts because CDD employees can add all of their updates on a given capability to the central database and R&D can choose to inquire about the things they find relevant to their problems. We must remain cognizant of this relationship and other ways that problems could manifest, but it does not appear to be as serious an obstacle as we once anticipated.

Interviews (10/92)

MSG Matt – C4I

• [examined the wire framing of our mockups]
• We need to be able to at least save as a link any kind of attached documentation. Ideally it’s drag and drop and gets stored somewhere else so it doesn’t slow down the search, you only go for that extra info if you want to deep dive.
• Thinks we may need a reminder system to get people to input stuff on their trips to “capability providers” could be as simple as checking a box on a Request for Orders form that we’d be collecting data from anyway.
• We are hoping at some point in the future to integrate this system with another we currently own the IP on that would let us use this data to populate a sort of capability Wiki.

MSG Joe – C4I

• [examined the wire framing of our mockups]
• My issue with the trip report submission page is that one comment box is too unstructured. I know it’s an informal process that went by the wayside, but I think this is an opportunity to at least loosely organize them into a few fields.
• Even something like “Who, What, Where, When, Why, How?” fields could be immensely helpful.
• Likes that tags eliminate the need for quite a few fields and enables splitting up the trip report a bit.
• Regarding Request for Orders forms prompting a trip report: it’ll help the commodity area people give a heads up on what a capability provider is like prior to the meeting. But it’s also insurance for the R&D person: if they posted that they were having this meeting and nobody from CDD follows up, the R&D person can claim they did their part.

MSG[?] Sandy – C4I

• [examined wire framing of our mockups]
• Small procedural point – sometimes trip dates change, so if you don’t ask after the fact then the dates might be wrong.
• Ideally if I don’t know who to talk to in a commodity area, I should be able to identify the commodity area itself and know that the right person in that commodity area will be notified.
• Category-wise, all of our capability gaps are getting funneled through our commodity areas one way or another, so at least getting the right capabilities into their commodity area buckets is very helpful.

Nic – Squadron R&D Deployment [Search MVP Test]

• Would want to be able to subscribe to keywords related to his commodity area and receive email updates on trip reports, but could see his inbox getting clogged up.
• Wants to be able to subscribe to certain requirements and automatically get updates on trip reports related to those.

MAJ Josh – R&D Section Leader

• Generally has a positive relationship with CDD. Sees them as primarily focused on R&D, while squadrons are also focused on operations.
• CDD makes life harder when they’re not in sync with what the operational guys want, or if we have to chase them down in those times where we do want their help.
• The incentive and disincentive against sharing is money – sharing can open you up to new sources of funding but it can also open you up to scrutiny and competing projects.
• Sees no downside to submitting a preliminary description of what he is working on before he really gets started – the disincentive against sharing in his opinion comes once time and money is already invested.
• His team needs status updates on ongoing procurement efforts, work on new procurement efforts, monitoring of emerging tech, and summary reports from trips.

Brian – Signal Squadron

• Confirmed interest in seeing any After Action Reports related to integrating new tech into systems he is working on.
• Confirmed interest in searching by a specific vendor and looking up all trips related to that vendor.
• Confirmed interest in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
• Confirmed interest in subscribing to specific existing requirements and getting updates about trips related to it.
• Confirmed interest in sharing articles related to a particular topic within the system.

Doug Brook

• In his experience, the problem wasn’t so much that people were never willing to change. It was that once they decided they didn’t want to change, it was impossible to move forward.
• Recalled a major overhaul to DoD’s personnel system. Army and Air Force were on board because they didn’t care much for their systems. Navy did not want to change, and Marines were particularly opposed because they had just overhauled their own system. The overhaul subsequently never really got off the ground anywhere.
• Did not recall a single time when he was able to force a stubborn organization to implement a policy. Always found the best way was to work with those boundaries as best as possible.
• Agreed with a bottom up approach – only working with the organizations that express a need for what we are doing helps reduce the chance of someone sinking the program, likewise for focusing on buy in from end users.
• Agreed that if our premise was correct that most organizations didn’t really have a good system for what we were doing it would be a real advantage in avoiding institutional resistance.

Erik Stayton – User Experience Researcher, Nissan

• Skeptical of people who claim to hate they computers in the sense that he doubts they specifically hate computers. Rather, they’ve never found them useful in doing their jobs. If they don’t see the value there, they’ll be reluctant to change, even if the current method is imperfect.
• So part of the effort will be to get people to stop worrying about the database, but part will also be changing what database means to them.
• They could also be afraid of the implied change of a new system, and might have some additional fears you’re not picking up on yet. But to some extent you can’t totally control this. If it’s a cultural problem, there’s going to need to be change regardless.

Nic – Squadron R&D Deployment [Input MVP Test]

• Would use the current tagging system to file his trip reports.
• Would also find value in being able to submit “pre-requirement” problem statements to get notified if a similar problem statement already exists and whether or not CDD is working on it.

SGM Katie

• Identified searching for external vendors to see who contacted them previously and tagging trip reports to notify any interested parties as high priority.
• Would want email notifications for relevant trip reports but not Requests for Orders – thinks there is too much variance in RFOs to get useful information out of them.
• Does not believe that log forms will help them identify “inappropriate” O&M purchases – there simply isn’t enough transparency with those purchases to identify remotely. It could help on a smaller scale within R&D sections though.
• Would also want to be able to subscribe to requirements, but doesn’t believe it will be as high priority as subscribing to and tagging vendors.
• Believes submitting informal problem statements would help as long as people self-identify when submitting so CDD can track them down and make use of such info.

Hypotheses Confirmed/Debunked

• Confirmed: Beneficiaries see value in searching and subscribing to After Action Reports. Our CDD beneficiaries see value in checking back to see which tech is actually being used, R&D beneficiaries often work on systems where other techs get integrated and would want to know of their existence.
• Confirmed: Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor. Different beneficiaries use this information for different purposes but consistently identify finding the appropriate trip reports as the best way of getting the information they need.
• Partially Confirmed: Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached. Several beneficiaries said this would be valuable information but did not necessarily confirm that they would be interested in submitting their own problem statements.
• Confirmed: Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it. Beneficiaries continue to look for an effective replacement for their current requirement tracker but don’t see requirements as high priority compared to the trip reports. However, the email notification system is perfect for this particular area, as they’re also specific enough that there was less concern about getting spammed.
• Partially Confirmed: Beneficiaries wish to share informal information sources such as news articles. Playing off some people’s suggestions that this kind of functionality be included, we asked some others who had made no mention of informal sharing in the past and they confirmed it would be helpful but were not particularly excited about it.
• Partially Confirmed: Beneficiaries want email notifications for relevant trip reports. However, they want some ability to be selective to avoid the spam problem mentioned above.
• Partially Debunked: Beneficiaries want email notifications for Requests for Orders. Nobody expressed a direct interest in seeing the forms. We have reason to believe that certain information that seems to show up consistently on RFOs could help but will have to reframe as nobody is looking at the forms themselves.
• Mostly Debunked: Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. People saw use for this in some of the tighter knit circles but for the most part didn’t think they could glean anything useful from the data that’s available on a Log Form.
• Confirmed: R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD. Beneficiaries clarified that the risk of friction between R&D and CDD exists in to main forms: when R&D has already invested time and money in a project and when CDD imposes certain ideas on R&D’s busy schedule. The former could potentially be addressed if CDD responds to informal updates quickly enough. The latter could be addressed through the system – CDD can put their findings in the database and R&D can take or leave what they find useful based on their specific context.

Questions for Next Week

Do the forms that beneficiaries expressed little interest in offer any kind of data that we could use consistently in our system?

How could informal problem statements work without overcomplicating the existing system?

Would CDD agree to use our system in a way that we think mitigates R&D concerns? Is it different from how they intended to use it?

Do any end users of capabilities outside R&D intend to use our program?

Can our existing functions be leveraged to accomplish some of the other goals mentioned in the original problem statement?

What will a proper, real world test of our system look like?

Questions for This Week

• MVP Applications: Is our tagging system effectively capturing key engagement details? Is our tagging input system intuitive? What information would beneficiaries submit using our form?
• Unit Dynamic: How can we ensure that R&D does not feel they are taking a major risk submitting their data to our system? What boundaries are necessary for each of our beneficiaries to accomplish their goals in this system? How do we achieve sufficient buy in across the unit to implement the system?
• Dual Use: Can people from other agencies understand the intent of our MVP? Can they envision how they might use it in their context, even if it’s currently built with USASOC’s workflow in mind?

What We Are Testing

• Our manual input form is intuitive and easy to use.
• Our manual input form only asks for relevant information.
• Our manual input form allows for the submission of all relevant information.
• Individuals who are highly concerned about CDD interference will not share any information about their projects.
• If the squadrons and CDD buy in to the project, apathy from higher ups will not be a problem when it comes to implementation.
• Beneficiaries seeking out a project have enough context to work with limited information. [condensing search output]
• Search and input MVPs are relevant to different kinds of workflows.

Key Insights

Our most important feedback this week was that our input MVP seemed to have the flexibility to cover a range of reporting that we did not anticipate. Our contacts in AFSOC and JSOC both cited ways they could see themselves using it despite that the MVP was not designed with their workflow in mind. This makes us more confident that we are striking the right balance between having some operational guidelines while providing considerable flexibility to submit whatever information is relevant. It also suggests that such a program could be functional across units without drastic changes.

Culturally, we learned from Nic in R&D that the core communication issue isn’t necessarily that people’s projects will get shut down. The real issue, in his opinion, is that people don’t feel like they’re being heard when they do talk to CDD and therefore don’t really see an upside to reaching out. Shutting down projects might not be a huge problem, but it’s still a downside, and the downside is what people will think about if they don’t perceive an upside. In theory this perspective fits with some of the prior testimony we have heard, but we would like to hear more about it from those who previously expressed concerns about CDD interference.

Interviews (10/82)

SGM Angel – Future Concepts

• Even among people who suspect CDD will try to shut down their projects, there is usually a level of information they are willing to share. Can’t speak to specifics for every individual.
• Affirmed the value of reaching out to Subject Matter Experts after submitting a proposal/requirement. It can be difficult to know if you have all the relevant information just from looking at a report, so the ability to check in is important.
• People can currently check the status of their pre-validation requirements via DecisionTracker. Most people hate it because it’s such a pain to access through the portal.
• Ideas get triaged all the time, particularly at the Requirements Review Board level. People need to accept that some ideas will not move forward.

Ben – AFSOC

• [on search MVP] It would be nice to somehow stitch together pertinent information from multiple site visits/trip reports so the customer could have a consolidated picture of past convos.
• A streaming comments section would help as well to make updates easy.
• Easy add: recency filter. We should be able to cut out stuff that doesn’t matter anymore.
• Harder add: relevancy filter; “Decision X has been made which renders Y information obsolete.”
• Wants to know how old information gets scrubbed so that the search function stays clean.

Brian – Signal Squadron

• [on input MVP] I was just on a trip that I would need to fill one of these out for.
• Thinks the types of inputs we are requesting would be easily searchable.
• Did not as easily envision the exact process by which the form itself would be submitted.

Doug Brook – Professor, Fmr. Asst Secretary of Navy

• [Inquired about the issue of siloing at the DoD level]
• The Pentagon has uncountable silos.
• Silos can be organizational, functional, or operational.
• As such, it’s difficult to provide an illustrative example of exactly how these divisions play out. Rather, it’s more accurate to say that each individual views whatever topic is under discussion through the narrow lens of their job.
• Leaders should theoretically bridge divide but there are many ways to hamper initiatives that don’t fit your interests.

MSG Matt – C4I

• [on search MVP] “Very [useful], I would likely use this at the outset of every project.”
• When researching, would use MVP to get a specific vendor’s history; any past interactions, any notes on performance or capabilities.
• Would also want to use it to get a broad overview of what we’re doing in a given technology area.
• Helpful deconfliction tool.

MAJ Mike – CAG in support of JSOC

• Suggested we look into Project Vulcan.
• More limited in scope than what we are doing as it mainly solicits information from vendors
• This functionality alone was enough to generate some buzz with his people; he found out about this program by chance through one of his teams.

Nic – Squadron R&D Deployment

• The whole building is visual, you have to be careful about adding too many layers to this thing.
• Doesn’t necessarily care about the timeline of when his requirements get validated. Just wants to know when the tech he wants will be in his hands.
• One element of the CDD/R&D issue is trust. People feel like they’re not heard when they do speak up, so there isn’t a lot of upside to sharing. Easier to think about the negatives.
• End users need to be able to focus on the mission. We either need to add non-operational people to the squadrons to focus on this stuff or have CDD do it.

Rob – CDD Operations

• Currently uses a computer system to submit RFOs.
• Would not intuitively know enough about his colleagues trips to fill in the gaps if a trip report wasn’t clear on, for example, whether they were visiting a particular vendor.
• All vendor related trips can be shared organization-wide. He does not suspect anyone on his team would object if he decided to share some of their reports.

CAPT Nick – Kessel Run

• Air Force going through similar issues as Army. Will deploy products 3-5 years after everyone stopped asking for them.
• Decisionmaking in the military is most frequently top down and this causes problems. Higher ups make decisions based on their experiences, which can be miles removed from the situation on the ground today.
• Believes that buy in from higher ups is still necessary but can be attained if you show the value the program has for subordinates. Everyone wants to look like they’re doing a good job, so is there a way that supporting the program will help him do that?

MAJ Mike – CAG in support of JSOC (MVP feedback)

• Liked the balance of available tagging options and flexibility to add new ones in input MVP.
• Would expect to see all engagements go through this system.
• Key info: who, what, where, when, why, and how to proceed.
• Also helps to know who set it up, the original purpose of the meeting, and whether the outcome aligns with the original purpose.
• Would be nice to be able to search the content of the trip reports.

Hypotheses Confirmed/Debunked

• Confirmed: Our manual input form is intuitive and easy to use. Most users seemed to understand the purpose of each tag.
• Partially Confirmed: Our manual input form only asks for relevant information. Nobody pointed out a particular tag as being irrelevant. However, nobody would use every tag on the list every time, so it’s hard to speak to what the unit as a whole would find useful.
• Partially Confirmed: Our manual input form allows for the submission of all relevant information. It seems the flexibility of the system empowers people to think about how they’d submit whatever form they’re thinking about. Our Air Force contacts discussed using it to submit forms that we were unaware of.
• Debunked: Individuals who are highly concerned about CDD interference will not share any information about their projects. There is some division as to exactly why some people are so worried, but this may be tied to the amount of information shared rather than whether it’s shared at all.
• Partially Confirmed: If the squadrons and CDD buy in to the project, apathy from higher ups will not be a problem when it comes to implementation. We will not know until we actually try. However, interviewees indicated the value of buy in from the ground up when getting a higher up’s approval.
• Debunked: Beneficiaries seeking out a project have enough context to work with limited information. [condensing search output] We found instances where people wouldn’t know things we would expect them to contextually know. However, people are also OK with getting a good contact an reaching out.
• Confirmed: Search and input MVPs are relevant to different kinds of workflows. Its relevance to our contacts from outside USASOC indicates it has a degree of flexibility.

Questions for Next Week

Is our input system more convenient than whatever people are doing now?

How many forms could our input system replace?

What amount of limited information is even the most paranoid R&D employee willing to share freely?

How can the search interface be condensed effectively? What kind of info should a condensed version display?

What value can CDD deliver that makes communicating with them worth the risk of getting a project shut down?

How can supporting our product benefit higher ranking people at USASOC, even if they don’t intend to use it?

Questions for This Week

• MVP applications: Given the desire to find a point of contact through our system and assuming we can provide this point of contact, what other information would be helpful to see right away?
• Unit Dynamic: In what instances has CDD ‘overstepped its bounds?’ Does R&D’s recollection of these actions align with CDD’s justificaton? Is CDD capable of providing some kind of assurance that it won’t directly interfere with R&D efforts?

What We Are Testing

• When using our MVP, beneficiaries can envision the type of information they would look for.
• Our MVP correctly displays the types of information our beneficiaries want.
• Our MVP currently displays relevant information in the most easily digestible form.
• If there were zero technical challenges to communication, some beneficiaries may choose not to communicate regardless.

Key Insights

Our most important feedback this week was that we are on the right track with our general idea. Our potential users and our problem sponsor were consistently able to describe ways that they see themselves using this system once it is set up. With that level of confidence in the core concept, we are now more at liberty to sort out the details that will allow us to consistently display the right information at the right time. At this stage, we were presented with two interesting challenges: how to contend with trip reports that cover multiple vendor engagements and how to track the problems themselves rather than just R&D efforts. We will need to better wrap our heads around those two issues before we decide what is viable.

We also gained some valuable insight on some of the perceived cultural problems we’ve explored for the last two weeks. In particular, Angel clued us in to the historical context of why and how CDD might shut down a given project and how that perception still creates challenges even though they’ve worked hard to change their behavior. Given the extent of testimony on the subject since our base visit, we are now in a good position to discuss what kinds of boundaries would minimize a beneficiary’s reluctance to contribute to a central database.

Although it was not our focus this week, our interview with Tim, who built a similar system for the White House, yielded an important insight into encouraging proper data entry. In his system, he created a field where people could list a proposed follow up date, then on that date he would go and ask if that person followed up. This let people know that someone was paying attention to what they were saying and made them more likely to do it properly. Consistent behavior on one end encourages consistent behavior on the other.

Interviews (10/72)

SGM Angel – CDD – Future Concepts

• R&D people look for solutions on their own because they can do it faster than CDD, even though CDD is faster than the rest of the army, and because they tend to be among the few with core competencies in the areas they’re looking at.
• The problem with this is that R&D people don’t always understand the acquisition rules, and CDD doesn’t always have insight into the problem R&D is working on, which can lead to potential rule breaking.
• CDD had a historical tendency to shut people’s projects down when it found out about this kind of rule breaking rather than communicating the problem. This is changing but the perception still exists.
• Believes that for people to freely share that info, they need to feel safe that they’ll be able to work on the projects they’ve determined to be important.

Dr. Jared – Course Adviser

• In our class some people ended up with a prototype because there’s an obvious need and can build something without actually working on their data.
• Sometimes the answer to the problem isn’t necessarily a product, but a process recommendation.
• They kept working on the project after, particularly on the technical approaches. They found it was technically possible but were never able to demonstrate at scale. He’s working on some of the same issues today.

MAJ Mike – CAG in Support of JSOC

• I understand that USASOC is more interested in vendors and products but I’d want to track all engagements. Anytime anyone has a meeting with someone, particularly at another government agency.
• With that in mind, I’d recommend making ‘trip reports’ broad enough to include information about any kind of interaction.
• You will need to contend with the fact that sometimes people have multiple engagements on one trip and make it easy to distinguish between different vendors/agencies/POCs. Otherwise you’ll get data about a few vendors listed under a single vendor.
• Setting aside the specific things I’m interested in, the features themselves seem like something I would use. I’d maybe want the email subscription to do some sorting for me.

COL Mike – CDD Director

• When I talked about scraping data from publicly available proposals, the main thing I was interested in is landing on the right person to talk to. I want to easily know what’s out there and can basically do that now, but it’s harder to find the right point of contact to learn more.
• What I imagine is being able to connect our problem statements to a database query and see what comes back.
• Even on projects that we already decided to spend money, it’s hard to get the people who pushed for those projects to follow up. That’s because deployment might be a couple years away and they have new problems to deal with.

Rob – CDD Operations (MVP Update)

• Liked the basic idea of the mockup and felt he could ID ongoing R&D efforts from it.
• However, would like to track R&D efforts “before they become R&D efforts.”
• In a perfect world, he could track problems as they arise, before they turn into R&D efforts.

CW3 Steve – Network Operations

• Confirmed that USASOC’s networks could support a python web server.
• Requested specifications on memory, storage requirements, etc. prior to implementation.
• They’ll also need some version information so their cybersecurity team can look everything over.
• Once the client is installed he does not foresee any problems.

Tim – Defense Digital Services

• Managed similar problem on White House networks, which wanted to have more modern forms of communication while complying with Presidential Records Act.
• White House also wanted to track tech engagements, had many instances of duplicated trips.
• The consequence of spending so much money on those engagements is they had to be more choosy about which tech they landed on, which is a problem when dealing with stuff that’s going to become obsolete pretty quickly.
• Key implementation tactic: included a field for proposed follow up date, and actually checked in with people to see if they followed up on the promised date. This convinced people that the data they are entering is actually being used.
• Key info for me: who reported it, date submitted, issues and comments, which other issues it’s linked to.

Will – Defense Digital Services

• Our ultimate goal is to put in place the right technologies to mitigate the risks from our adversaries in any possible situation.
• The tech itself is easy to find, the challenge is applying it so that it works for us, then getting it deployed.
• Our team of about 45 consists of engineers, product managers, and ‘bureaucracy hackers.’ The engineers figure out how to make stuff work, the product managers figure out what combination of things we want to use, and the bureaucracy hackers make sure we’re compliant with the law.

MAJ Tony – AFWERX

• Confirmed that our MVP appears to be attempting to solve the correct problem.
• Suggested that civilian companies may have similar issues.
• Increasingly concerned about the input side – people will absolutely have to input data for the system to work but people are lazy.

Nic Vandre – Squadron R&D Deployment

• Described himself as an end user in the field, saying he needs a capability. His role is to take new tech and figure out how it will work at the warfighter edge.
• He gets his tech from two places: whatever CDD has for him and whatever he can find in industry.
• He won’t tell anyone about his projects until he is ready to submit a requirement. Sees the unit as highly autonomous and low on vetting; one person submitting requirement can hold a lot of sway. He primarily vets ideas with his squadron.
• Considers his industry research to be supplemental to what he sees as CDD’s job. Believes CDD should be on top of what industry is offering and keeping him posted on what they’re going to send.
• In general would like a better way to track how CDD is responding to his needs and whether they intend to follow up.

Hypotheses Confirmed/Debunked

• Confirmed: “When using our MVP, beneficiaries can envision the type of information they would look for.” Beneficiaries seem to have a general sense of what they are looking for to solve their problems when they look at the system. Some people are looking to associate a specific name with a given engagement so they can follow up in person. Others want to see what recent activity is going on around a specific type of capability to try to anticipate needs that will take up some portion of the budget. Others want to make sure nobody is working with the same vendor as them or on the same capability as them.
• Partially Confirmed: “Our MVP correctly displays the types of information our beneficiaries want.” Generally speaking, the information that is already on display in the MVP is seen as relevant. However, some people worried that our system wouldn’t do a great job accommodating the existing habit of reporting on multiple engagements in a single trip report and would want to know that the system is attempting to address that on some level. We also heard from multiple people that they’d like to be able to track *problems* and not just what capabilities people are looking into. Rob in particular felt that it would help him in his job if he could see the problem before R&D undertakes any efforts to address it.
• Debunked: “Our MVP currently displays relevant information in the most easily digestible form.” This hypothesis was primarily debunked with the help of our problem sponsor, who gave us a better sense of the volume of entries he’d expect on such a system. In essence, our existing method of displaying the text of the trip report would become overwhelming if people start doing searches that yield hundreds of results. Would instead like to see an easy to scroll through list that he can expand/collapse as needed.
• Confirmed: “If there were zero technical challenges to communication, some beneficiaries may choose not to communicate regardless.” Discussions with Angel and our problem sponsor confirmed that there can be a real fear among some people submitting their projects. Usually because they think their project is the most important thing, and the last thing they want is some higher up who doesn’t know any better coming in and making changes or even shutting it down. Joe suggests there is a gap between fear and reality.
• Partially Debunked: “CDD goes around shutting down people’s projects without knowing what they’re doing.” Joe offered some clarification on this front, suggesting that the typical issue when something needs to be shut down (maybe just temporarily) has to do with complying with acquisition law. Angel supported that assertion in her call but said historically CDD maybe did have a problem with identifying a rule breaker and just shutting the thing down without communicating the problem and trying to have a discussion. She has put in significant work in the last 10 years changing that behavior but thinks that the perception is much harder to shake, even though a lot of the R&D people working now weren’t around back then.
• Confirmed: “Government agencies similar to USASOC could repurpose our MVP to meet their specific needs.” Tony and Mike both suggested that our MVP was rather vendor-centric given their needs, but saw no reason why the engagements they are interested in couldn’t be included on a more generalized form as long as people are willing to report it. In essence, they saw themselves as similar enough to make this kind of solution viable.

Questions for Next Week

• MVP Applications: Is our tagging system effectively capturing key engagement details? Are we pulling data from the right places? What kinds of manual input do we expect and how will they do it? Is there a simple way for this application to track “problems” rather than just capabilities?
• Unit Dynamic: How can we ensure that R&D does not feel they are taking a major risk submitting their data to our system? What boundaries are necessary for each of our beneficiaries to accomplish their goals in this system?
• Dual Use: Can we get anyone from private industry to talk with us directly about this issue? Can we generate signs of interest on our own?

Interviews (10/62)

MSG Tom – Future Concepts – Subject Matter Expert

• Not sure what his most common search terms would be to find people who visited a given vendor or worked on a given capability.
• The information that eludes him, and that he’s most interested in, is identifying previous colleagues that had visited a specific partner, whether that’s another government agency, lab, or vendor.
• Would also like to know about what any previous colleagues had discussed but would rather find out with a quick phone call than by going through piles of documents. Going through piles of documents is what he does now and he’s never confident that he has all the information he needs.

Rob – CDD Operations

• If he’s going to search for something, it will probably be by vendor, category of equipment, and/or timeframe.
• It can be helpful, for example, to know who talked to a given vendor in the last 6 months. He should be able to easily distinguish the timeframe he cares about from no longer relevant entries.
• The categorical search helps when he wants to know what vendors people are talking to about a given type of product.

MAJ Mike – JSOC

• Emphasized the value of a system that is both simple and able to put information right in front of people’s faces. Felt our email subscription solution checked those boxes.
• Wished that it could somehow use the data collected to show who is due to provide a trip report. Noted that similar products are briefed all the time in the military and can be a good motivator. Did not provide an example of a time when such a system proved effective.
• It would help to have some basic templating to help folks who are totally unsure what to say to write something. Stuff like date, participants, products, conversation, and the ability to attach a file.

John – Squadron R&D Specialist

• Has experienced both visiting a vendor only to find out that someone else had already made contact and finding out that someone visited a vendor he was already working with.
• Planning on a trip next week – would absolutely search a database to see if anyone else made the same trip. Trips cost time and money.
• Believes problem can extend beyond duplicated and is reflected across other government agencies – individuals can spend years working on a capability only to find out that another agency already developed it.
• Warned of a slight distinction between how civilian employees apply to go on a trip compared to enlisted employees. Each form includes the same basic information but may require looking in different places.

MSG Darrell – R&D Section Leader

• Estimated he spends about 10-20% of his time interacting with CDD but it depends on the project. For some it’s 100% involvement, for others they might not need any.
• Value’s CDD’s understanding of the acquisition process; suggests they may sometimes overstep their bounds in shops that are not part of the units core competencies (which he considers to be the more traditional development teams like such as guns, breaching, etc.).
• Suggested a willingness to share information about his projects at an early stage with two caveats. First, this already happens among the more tech savvy people and CDD is not often a part of it due to lack of right personnel or passion. Second, people in his position would worry about CDD using such a platform as a way to wield oversight over them and shut projects down.

Brian – R&D Section Leader

• Noted that his team primarily receives new capabilities and has to figure out how to make them work with all the existing systems.
• His problem: with the increasingly complex layers of networks he has to work with, it becomes increasingly difficult to cope when he suddenly finds out he needs to implement a capability that’s been in development for years.
• He wants to know anything new happening with ATAC (their primary software interface). For example, when someone uses or integrates a program in a new way, right now they’ll write up an AAR saying what they did, what they’re going to keep doing, and what they need to improve on. He doesn’t know where they go. Once they get emailed if he doesn’t see them right away they might as well go in the trash.
• What he would REALLY love is to be able to put the content of webpages onto a program where people could comment with their work related details. He follows a number of national labs and believes they provide the best info about capabilities, but it’s extraordinarily difficult to get that type of information on the classified network where they do most of their work.
• Indicated that he tries to keep CDD in the loop on his projects and generally sees CDD as helpful in making the money work for various projects.

MSG Tom – Future Concepts – Subject Matter Expert (MVP Feedback)

• Liked that all the data would go to one place, but noted that his problem isn’t finding data, it’s finding the most relevant data. He can go through a squadron’s trip reports be he has no way of telling which one is relevant to his needs.
• Clarified that he wants to be able to call people because he’s also interested in knowing what was left off the trip report. Finding the people who talked to a specific vendor or agency is a 10/10 score in his book. If he gets a name, he can make a call and know what he needs to know in five minutes.
• Sees further value in attempting to categorize the capabilities they are working on and search by initiative type. Sometimes teams with completely different specializations wind up working with the same vendor.
• Values the ability to find no results and as a result say with reasonable confidence that nobody else is working on a similar capability.

Rob – CDD Operations (MVP Feedback)

• Only cares about 4 categories of trips: visits to vendors, visits to trade shows, visits to demonstrations, and visits to test capabilities.
• The majority of trip reports are useless to him and he does not want to have to sort through them to find relevant ones.
• Would want the ability to do some level of filtering by type of trip in order to use such a program.

MAJ Tony – AFWERX

• Unit has 12 full time staffers and as many as 100 people working on different platforms nebulously working as part of organization.
• Thinks traditional KM solutions like Salesforce need an admin to be sustainable.
• Primary communication challenge is uncertainty over who to connect new entrepreneurs to within organization.
• Suggested the problem in USASOC is also present in AFWERX, DIUx, and Cyberworx. Believes AFWERX would take notice if one of those organizations could successfully address it.

CAPT Joey – AFWERX

• His job right now is essentially to solve our sponsor’s problem on a small scale within his team at AFWERX. Believes the next great challenge is finding a way to scale tech scouting effectively.
• They are currently working on the problem from a user adoption side and from the side of pulling data they want not just within the unit but across DoD.
• In a perfect world they could see all the thing in the pipeline not just within their unit but across other agencies and the commercial sector.
• Primarily believes this is an issue in the burgeoning venture capital space within the government (AFWERX seems to function more like a VC group than USASOC).

Hypotheses Confirmed/Debunked:

• Confirmed: Beneficiaries within R&D and CDD want to be able to easily discover and contact people who are working on similar projects. Respondents suggested they already take time to do this and would gladly use a system that shortened the process.
• Debunked: Beneficiaries know exactly how they would search for technology relevant to them. Only one respondent suggested a specific type of capability he would subscribe to. Another suggested that the challenges associated with searching undermines his confidence that he has the best information available.
• Debunked: If all relevant data is in a central location, people will more easily be able to find it. Tom’s testimony suggests that if the tool cannot reasonably parse through a large volume of data and identify the most relevant results to him, it wouldn’t necessarily matter that he was able to do it all in one place. He still wouldn’t have confidence in his findings.
• Confirmed: If we build a program that benefits USASOC, that program can provide similar benefits to other tech-centric agenci
• Confirmed: The gap between R&D and CDD is more than a failure to communicate. After weeks of reading between the lines, we finally heard some testimony indicating that some parts of R&D see CDD as intrusive and potentially threatening to their projects. Now that we have some evidence that this tension exists, we will need to better understand each side’s perspective so we can better account for it implementing any solution.

Questions for Next Week

• MVP applications: Given the desire to find a point of contact through our system and assuming we can provide this point of contact, what other information would be helpful to see right away?
• Unit Dynamic: In what instances has CDD ‘overstepped its bounds?’ Does R&D’s recollection of these actions align with CDD’s justificaton? Is CDD capable of providing some kind of assurance that it won’t directly interfere with R&D efforts?
• Dual Use: Are other tech-centric government agencies similar enough to USASOC to effectively implement a USASOC solution? Is its value broad enough to generalize to government as a whole? To commercial industry?

Interviews (10/52)

MSG Edward – Network Operations

• Available databases include: rudimentary Request for Orders forms, inventory management, CDD purchases (buy try and procurements), O&M purchases.
• Every penny of government money has a receipt. There is a document somewhere proving someone spent that money.
• O&M Log Form request goes into a database as soon as you start the process, before it is approved. Currently housed in Microsoft CRM but working on a way to integrate with financial system.

Cory – Deputy Dispersing Officer – Finance Office

• How travel works once RFO is filed: First, organization signs off on travel. Second, comptrollers assign account lines to rfo order with an estimated $ amount for trip. Third, HR creates the document (1610 form) of what you’re allowed to do/where you’re going. Many people don’t take those orders with them.
• You go to the location, do your thing, come back, use your 1610 voucher to fill out a 1351-2 voucher. That includes itinerary. Someone reviews and validates that, then it comes to finance office with your receipts and your orders.
• Plugged into system that automatically tabulates how much you are owed for the trip.
• Information shared on RFOs and 1351-2s do not necessarily include all info about trip.

Ed – Network Operations – Travel Voucher Portal

• Process handled on system called IATS. The owners will not share data necessary for integration and they can’t get another program due to red tape and cost.
• IATS is also a standalone server in the building, part of a system used throughout DoD. It can communicate with DFAS but will be hard to connect with otherwise.
• Unclear whether the primary obstacle is DFAS unwillingness to share or lack of cooperation from vendor. However, IATS was apparently an off the shelf solution (fascinating in its own right) and we suspect that this could be causing complications with current attempts at customization.

Tim – Higher Echelon

• Salesforce intent on building high level reports from data aggregates.
• Focus on capabilities rather than value provided.
• Strategic focus suggests that an entirely off the shelf solution isn’t feasible.

Natalya – Salesforce – Developer

• Uncertain how long it would take to integrate Salesforce UI into USASOC networks.
• Uncertain which data sources Salesforce would use to generate reports.

Sean – Salesforce

• Demoed potential product for us.
• Emphasized ability to track money spent with a vendor over time – unclear whether it could distinguish funding streams without additional input
• Lots of features packed into a sales oriented interface. Lots of little potential solutions that can be overwhelming when taken in all at once.

MSG Joe – C4I – Commodity Area SME

• Highly invested in finding a more efficient way to track projects – the time he spends trying to find people is time he could be spending looking into the tech.
• Insists that despite the idiosyncrasies between different teams, the core organizational workflows are well defined.
• Strong believer in leveraging existing data flows, as opposed to requiring new forms of data entry.

Gabe – Squadron Ops – Built an Engagement Tracker

• Confirmed interest in tracking “RDT&E-like activities” and that it would aid in the deployment process.
• Engagement tracker worked initially but fell out of use as people left.
• Believes there were three key problems with the program: maintaining awareness of its existence, the presence of a learning curve to use it, and the inability to turn it into an institutional behavior rather than personality reliant.
• That’s the main problem, we sometimes get the right personalities to deal with this stuff on a human level but no good institutional method.
• Noted that there was also a compartmentalized list of people who could see, use the program.

COL Mike – CDD Director

• Noted something that everyone has to do when they’re filling out a requirement – justify that there are no comparable solutions currently out there. It takes effort to do it right, most people just call a contact or two. But if they could make an effective case easily they probably would.
• Greater challenge than tracking vendors is which agencies are actively contracting with them. From my position I want to know if DARPA is working on the same thing as me. Some vendors are unscrupulous and will charge 3 different agencies for the same product without telling them what the other is doing, this at least gives us some leverage.
• Unfortunately, particularly with R&D, there isn’t a ton of incentive to share the program. R&D folks aren’t rewarded based on deployment and worry their funding will get cut if they are found to be redundant.
• If I want to see what industry is doing I can look up BAAs or SBIR initiatives and see who responded. It is much harder to find out who talked to a given company even though that data is publicly available.
• We have tried a number of CRM solutions to solve the issue of communication within the unit but it’s hard to get buy in. Would probably be more helpful to scrape that stuff automatically even if it’s not perfect.
• Sometimes people will misrepresent what they’re doing on an RFO because they don’t want you to know who they’re talking to. However, most people are just lazy.
• Everyone has to contact the vendor at some point if they’re gonna work with them, and probably file a request to visit, bring them on base, or even talk on the phone. There is some record of it at some point, even if it’s just an email in someone’s inbox.

SGM Rob – CDD

• Better clarified why informal knowledge of potential procurements would help – the budget is planned by the start of the fiscal year and anyone who comes in with new requirements after that screws up the process.
• Even worse, sometimes the way they develop the project with vendors means we have no choice but to find more money.
• The types of purchases that go through O&M aren’t necessarily the biggest threat to fly under the radar in this regard. We have some ways of tracking activity that involves an O&M request. Sometimes R&D-like guys get free samples that don’t require any kind of O&M form and decide they want to buy it, realize that they can’t do it with a log request and then take it up to procurement way later than they should have.
• We literally wouldn’t mind if people came and talked to us before they even started on a project, but there’s no forcing mechanism to get people to give us a heads up as early as possible.

Hypotheses Confirmed/Debunked

Debunked: “There is no mandatory, standardized, centrally deposited form outside of the requirements process that could indicate a future requirement.” While we brainstormed ways that less uniform methods of reporting such as trip reports and AARs could be leveraged to better track projects, we worried that we did not have a great way to account for the ‘failure to report’ problem due to the existing culture around these nonmandatory reports. A few beneficiaries at our base meeting and MSG Edward confirmed that all O&M purchases have a paper trail that starts prior to the release of funding. Properly tracking O&M expenditures to better predict potential procurements presents its own set of challenges but also an opportunity we didn’t previously think we had. We also had our original impression of RFOs challenged by Rob, who said 99.9% of the time they’ll get filled out before a trip happens.

Debunked: “All data at USASOC is readily accessible if you know where to look.” Database accessibility can depend on which networks it exists on and if it feeds back to another agency such as DFAS. It could get more difficult if we need to be able to interface with those systems at all.

Confirmed: “Most CDD budget experts can use informal knowledge of a project to appropriately plan its funding.” The real challenge comes after the budget is set because the money everyone is aware of has already been spent.

Confirmed (CDD side): “CDD budget experts believe that knowing about a project in advance will speed up the deployment process.” If a project gets reported early enough and doesn’t get funded, it’s probably not that high of a priority. So for CDD it clearly speeds the process up. Less clear is whether the time difference is meaningful to anyone outside CDD. COL Mike added that it increases the likelihood that someone at CDD can find a source of funding at another agency as well.

Somewhat confirmed: “This advance knowledge is valuable enough that CDD budget experts would seek it out if it were readily available.” It is clear from our conversations that people at CDD will go to great lengths to find this knowledge. However, that willingness does not seem to translate to a collective effort to get this information onto one of the many solutions they have tried. It seems something is blocking them from getting scaled across CDD.

Next Week:

– Do R&D cell leaders believe that CDD having advance knowledge of their projects will lead to faster deployment?

– If R&D cell leaders believed that providing CDD advance notice of projects would improve their deployment timeline, why don’t they always do it?

– Do R&D leaders associate any risks with sharing information about their project early in the process?

– If R&D leaders could be convinced that early reporting would speed up the CDD timeline, would they see value in that?

What level of detail is mandatory on RFOs? What data can’t consistently be collected from that form?

 

Interviews (17/42, 21 total for weeks 2 and 3)

Tim (Salesforce)

• Believes users will enter data if they believe it will make their life easier.
• Thinks unified adoption will require strong top down leadership.
• Believes it is advantageous when as much data remains unclassified as possible.
• Salesforce has some experience working with other SOCOM divisions but has not fully completed FedRAMP.

Natalya (Salesforce)

• Cost of this type of program is open ended; usually requires modification and government cloud is more expensive.
• Uses iterative model involving weekly feedback from end users.

John (DARPA)

• DARPA experiences similar info management problems.
• He is constantly approached by vendors offering new knowledge management features.
• Different groups will have different tolerance levels for doing work to make your solution function but are unlikely to want to do more than they currently do. The F-35 team for example is accustomed to a large paper trail.
• Solution should function with the organization as it currently exists and not depend on the organization changing.

General Dempsey

• Money prevents collaboration. People want funding for their own projects.
• However, this isn’t necessarily a bad thing. Friction between these groups is what moves the most viable projects forward.
• Friction is intensified when budget is tight.

Fort Bragg Interviews (ranks need to be confirmed, units are correct)

Rob – CDD

• Biggest pain point – finding out about an unexpected procurement after he’s already completed his spend plan. It’s very hard to change things once the plan is submitted.
• Believes part of the color of money reporting problem is the parameters around each kind. CDD should get a notice for any O&M purchase.
• We need to simplify the rules in a way that delivers our core informational needs.

Tom – Future Concepts

• We want to plan for when capability development is successful. If they fail it’s a waste of money but when it’s a success it’s almost worse because we have to decide to cut other things.
• When that kind of conflict occurs we have to go through a Requirements Review Board process that delays deployment.
• We need to be able to ask people whether they want those sudden requests or the $X worth of things we’re going to have to drop because of it, whatever’s at the bottom of the priority list.

Dave – A Squadron

• When I recognize a problem, I want to know my parameters. I want to know who has already looked at it, get a basis for where to start.
• Every squadron produces SITREFS. How can I get a notification when one comes along that is relevant to me?
• When I’m looking, I want to know who to talk to. Why didn’t it go forward last time? Do I need to revalidate?

Joe – C4I 

• We want to protect the unique authorities of the unit command. We don’t want to get in the way of what end users or doing but it’s necessary to protect that authority.
• We also want better efficiency in project management. The effort it takes to find all of these records could be improved considerably.
• We also want to be able to sort by commodity area.

Herb – CDD

• Who is the vendor/organization?
• Lead point of contact in unit
• market research
• Was conversation in person or remote?

John – CDD

• I’ve been here since 1995 and at that time we filed trip reports as a matter of discipline.
• It fell out of practice because people couldn’t really put them to use.
• To get people to submit them we need an efficient way to do it, a place to store it, and a way to access it for all users.

Tom – Future Concepts

• It’s good to save time but we also need buy in from leadership.
• Those trip reports aren’t for me, they’re for whoever is going to read them. So if I don’t think anyone is going to read them, why would I write them?
• I have no preference to exactly what form this system takes, I just think there needs to be a standard.

Brian – B Squadron

• I would argue that the strength of the organization is having a sandbox with parameters.
• Overstandardization could undermine why we operate this way in the first place.
• End users like giving feedback, they don’t like having to go to a meeting to do it.

Angel – Future Concepts

• Part of the problem is lack of planning – how much HW are you doing before you go on a trip?
• Even without a database the organizational knowledge exists.
• People know where to go if they want to talk to an expert, they just don’t do it.

Jason – Squadron ?

• I love tech solutions but I think this is partially a people problem.
• Like a squadron might need a bunch of censors and not even consider talking to signal squadron.
• They also don’t talk to CDD because they think it will slow them down.

Daryl – Intel(?) Squadron – End User

• We’re a bunch of highly motivated introverts. We have to get into the habit of talking to each other.
• If we don’t feel like CDD’s subject matter expert is helpful we’ll go around them.
• It would be great to subscribe to a feed that alerted me on upcoming trips, etc.

Pat – Signal(?) Squadron – End User

• I don’t always know who is going to be interested in what I have to say.
• I’ll send an email to Daryl and later in the day it’s blown out to 30+ people.
• If those people can find a way to request that information from me then they can get it even if I’m not aware of them.

Tony – C4I

• End users will do some work if it’s easier than what they’re doing right now.
• If it’s harder than email don’t even bother.
• Greater challenge will be connecting disparate silos of data.

Hypotheses Confirmed or Debunked

• Confirmed: Late notice of a project set to be deployed affects CDD’s ability to deploy its budget effectively. Late procurements may force CDD to defund other priorities or take the matter to a review board, which delays deployment for both parties.
• Confirmed: “Unofficial” knowledge of an ongoing project gives CDD a chance to plan its budget accordingly. CDD is not guaranteed to find the money needed but has more time to look for funding from its own lines or from outside agencies spending for a similar purpose.
• Confirmed: R&D leaders see some risk in interacting with CDD. While the sentiment was not universal, there is some perception that they will be slowed down or denied.
• Debunked: Trip reports are the standard means of reporting progress. We heard many references to SITREFS, with a couple even citing it as the reason they might find out about a given project before they are told.
• Debunked: End users aren’t interested in providing feedback. They are interested but hate having to go to a meeting to do it. As such, only those who really want to see a capability deployed will show up. More simplified feedback systems have worked better.

Questions for Next Week:

• Can we build a timeline out of our paper trail? Can we make reliable assumptions about the order in which our milestones would appear?
• Could certain O&M transactions be detected that would forecast a potential future procurement, and would CDD be able to parse such information effectively?
• To what extent to individual teams rely on their own way of doing things? Would they prefer a standardized system? What are they not willing to give up under a new system?

This week is base visit week and given our discoveries about the size of the workflow last week it has given us an opportunity to make sure we truly understand the organization, the processes they are concerned about, and key differences between the various groups that fill similar functions in different commodity areas.

Customer Interviews (4/25)

CW3 Steve – Network Operations

• Organization is currently working on boosting cloud offerings, including an unclassified cloud that could facilitate collaboration
• However, there is no organizational standard on where or how to keep data, which may not be fixable.
• We do much more work on the classified network because it is easier to share internally and there are more capabilities.
• It’s possible to get a vendor’s info on a classified network but everyone has a different process.

MGSGT Chad – Squadron Ops SGM – Funding for In-Unit R&D

• Not embedded in R&D section but supervises R&D section in Squadron F. Coordinates all of F’s activities and maintains continuity between projects.
• We have a unique mission set and receive special funding from Congress as a result. We can use this budget to tell CDD what we want.
• Each squadron has its own money.
• Individuals highly specialized and can’t necessarily cover each other’s fields effectively.

Ben Fulton – AFSOC

• AFWERX has a better view of vendor databases and which are worthwhile.
• Enables you to see what other Air Force Units are submitting similar problem statements.
• Sees some policies and regulations as getting in the way of acquisitions, but thinks it can be worked around.
• Pointed to an entrenched mindset of “Rapid innovation threatens job security.”

Tom Williams – Commodity Area SME – Communications

• Our job as commodity area SMEs is as enablers of capability development. They look at mission needs, capability requirements, and derived requirements and take the development process all the way to the point of obligation of government funds.
• Wishes end users would understand that taking a little extra time to go through CDD could allow for so many more resources and will ensure that those requirements are sustainable. Suggested that some users take dangerous shortcuts to cut a small amount of time away from the cycle.
• Believes some kind of focal point for end users where they could be put in contact with the right collaborator could be helpful, believes the benefit would be for the end users rather than CDD.
• As a general rule, believes CDD to be doing quite well (particularly given his experience in the Navy), and believes that the key is better communication with end users and convincing end users that taking a little bit of extra time to do procurement properly will be better for them in the long run.

Hypotheses Supported:

• R&D Sections are in charge of early, “unofficial” testing, CDD takes the requirements generated from that stage and eventually brings them to deployment. Our revised hypothesis on organizational structure remains intact following this round of interviews and it would appear we are on the right track of capturing who does what.
• R&D sections have flexibility to act quickly; CDD relies on requirements to act but has more legal authority to enable official R&D and procurement. Our hypothesis on the ‘characterization’ of these groups continues to hold water as well. Of particular note is fairly consistent testimony from CDD personnel that if end users could just be patient and not try to circumvent the acquisition process, there wouldn’t be nearly as much to worry about.

Hypotheses Debunked

• Getting information onto classified networks makes it unreadable by a program. Our interviews with some of the database experts at the organization revealed a number of methods for safely getting information on a classified network, including a system that allows vendors to send information directly. The problem is that most people don’t use them.

Questions for the Coming Week

• Why isn’t the CDD process perceived as fast enough? Are there any examples of CDD slowness compromising a mission?
• Who among the end user squadrons makes the decision to circumvent CDD to obtain a product? Is this shortcut only possible with O&M money and the micropurchase threshold or are there other ways?
• Is there any level of distinction between CDD’s yearly budgeting and its process of deploying tech to meet a particular requirement? (i.e. What exactly do they achieve in the budgeting process and does it give them the leeway to obligate funds toward some new requirement during that year?)
• Do we have the workflows and org chart right? (During our base visit we will be sharing some workflows and our organizational chart with our interviewees to give them an opportunity to correct any misconceptions we have.)