Questions For This Week

What should our combined interface include?

What is the range of opinions on what the division of labor between CDD and R&D should be?

What would a test run of the program look like outside USASOC?

Who among our existing contacts can help us with another kind of test run?

What We Are Testing

• Our input template can cover USASOC’s range of input types.
• Our combined interface is navigable and easy to understand without much prompting.
• Organizations outside of USASOC would want to test this program.

Key Insights

Regarding division of labor, Dave from Squadron H offered us our most valuable insight. He noted that end users real pain point is having to keep pushing a problem in the unit when they have things in the field to worry about. As such, they’re going to reach out to whoever they think will be most responsive. For any particular division of responsibilities to work, it will need to include a timeline that involves communication with the end user at certain key points in the process. R&D section leader Josh suggested that CDD should at minimum always be in the room and ideally always in charge of talking to vendors. The big problem with other people doing it is they don’t have the training to make sure they aren’t accidentally committing the unit to a purchase that they’re not actually committed to making.

Regarding our interface, our most valuable insight was to minimize the size of the blank spaces that go with our prompts. If we deliberately ask for concise answers to questions, we can simultaneously cut down on fluff and be less intimidating to the folks who aren’t big on typing.

Regarding outside opportunities, our most valuable insight came from Chris at BMNT. He suggested that the kind of problem we are working on comes through H4D and BMNT’s doors all the time and that he could even think of a project they’re working on where it may be viable to test our program. This is particularly significant given the range of DoD and federal entities that BMNT works with; if our program turns out to be a viable solution for the kinds of problems they are talking about then it would be a significant validation of the value we bring.

We are continuing the discussion with our problem sponsor about what deployment might look like but there have not been any significant updates to that process since last week.

Interviews (10/112)

Nic – Squadron R&D Deployment

• Would prefer pre-RWS submissions be submitted via an emailed link.
• Sees it as no different from submitting a visitor request, which people already do all the time.
• Difference is a pre-RWS submission wouldn’t be mandatory so people would need to be motivated to do it.

Mike – CAG in support of JSOC [Almost Ready MVP Feedback]

• Regarding filling out forms, likes the idea of light templates with prompts but thinks they should all be optional.
• Emphasis should be on concise responses, maybe even phrasing each template as a question could help with that.
• Ideally there will be a handful of pre-built templates to manage all the different forms, but also the ability to create a new template.

MAJ Josh – R&D Section Leader

• Believes CDD should be involved in potential procurements as soon as possible.
• Ideally, nobody talks to a vendor except CDD or at least CDD has reps in the room during the discussion.
• CDD personnel receive special procurement training to avoid talking themselves into an unauthorized commitment by the government. Your average Joe in the unit has a higher chance of getting themselves into trouble.
• CDD also has more of a birds eye view of what the organization needs – they are receiving requirements from everyone so they can help synchronize efforts and reduce duplication better than on our side.

MSG Joe – CDD – C4I

• Sees relationship between R&D and CDD as continuing to evolve.
• Believes he has a solid understanding of how things should work in his areas.
• Not certain about the “ideal” division of responsibilities.

Dr. Jared – Course Advisor – BMNT

• Deploying software without it being maintained isn’t really a viable solution unless you can get people to volunteer to look after it.
• One of the best reasons to start a company is to make sure that people actually use this thing and not let it languish.
• The only other option is giving the customer full ownership.

MSG Dave – H/A Squadrons

• The big problem regarding division of labor is that people get into their own little bubbles and have the people they contact who they think will get the job done, myself included.
• There isn’t exactly a consistent designation of roles, so when we do find a unique solution to our problem elsewhere in the unit it’s an extremely lucky break.
• A big pain for end users is that they don’t have the time to constantly push a capability gap – they want to send it to whoever will get the job done and get back to other things.
• People tend to get caught up in their own projects, which can divert the timelines on important capabilities.

Chris – BMNT

• Excited to hear about our project – many of the problems that come through BMNT’s door have to do with knowledge management.
• There may be an opportunity to test the program with a real problem they are working on.
• Similar situation profile: relying on email is causing problems. People don’t know who to send things to, there are limited number of mailing lists, and people are afraid to talk about niche things because of the spam factor.

Brian – Signal Squadron

• Doesn’t necessarily think he should be involved in every meeting, nor does he necessarily have the time.
• That said, he would like to know about stakeholder/face to face meetings as early as possible so he can adjust his schedule if he is interested enough.
• People have no problem injecting themselves into the situation when they find it relevant, but they need the time to find out and adjust accordingly.

Mike – CAG in support of JSOC [updated MVP]

• Emphasized limiting the amount of blank space on the forms.
• Feels this will be a psychological advantage for folks who are averse to writing.
• It would be nice to be able to search the answers in the prompts in addition to the tags.
• Thinks the focus needs to be twofold at this stage: bringing down the “cost of participation” for trip goers (i.e. time taken to fill things out), and making sure the input is useful for analysis and available somewhere in HQ.

Archie – NSTXL

• Current job is to build up his capability to network different H4D teams together so we can share best practices.
• He is meeting with a team called “Lumineye” soon that could be of interest to us.
• H4D’s most prominent success story is a company called Capella Space. However, it should be noted that they were already a startup before the course and primarily used it to further their existing goals.
• Wants to connect us to post-H4D capabilities as they get built.

Hypotheses Confirmed/Debunked

• Confirmed: Our input template can cover USASOC’s range of input types. Our current set up appears to be sufficient to gather relevant information on a variety of activities and could even ask for more concise answers, assuming that we can provide a few different versions of the template for the most standard inputs. This may create its own interface challenge as we need to make sure a range of input templates would not become overwhelming. However, it appears we are on the right track in terms of looks and how much info we are asking for.
• Partially Confirmed: Our combined interface is navigable and easy to understand without much prompting. Thus far, everyone we have talked to about the latest MVP seems to understand the functions. However, they are also people we’ve been talking to about this project all semester, so it remains unclear whether someone who is unfamiliar with the program would be able to jump on and figure things out right away. We will need to find a few people who are totally unfamiliar with what we are doing to really confirm this hypothesis.
• Confirmed: Organizations outside of USASOC would want to test this program. Chris from BMNT suggested that they work on knowledge management problems all the time and he could think of a current project where our solution might be applicable. No test plans with other organizations have actually been confirmed, but this is the strongest indication yet that we will have other opportunities to deploy the program. We are still in contact with Tony from AFWERX, but it seems the kinds of tests he had in mind involved us linking up with BMNT or another knowledge management firm so it is unclear whether AFWERX itself would want to help us test.

Questions for Next Week

What is USASOC’s perceived deployment timeline for this program?

Is our interface understandable to someone who hasn’t spent months in contact with us?

What would deploying on BMNT’s project look like? When would they expect it to happen?

What steps do we need to take to make sure we are prepared to manage relationships outside USASOC?

Questions for this Week

Can we fully confirm/debunk our hypotheses from last week that required more feedback?

For the forms that were confirmed useful – are we able to take the steps necessary to actually pull that data from its source?

What will our first deployment test look like?

Who will help us deploy the program?

What other opportunities exist to deploy this program in other units?

What We are Testing

• Beneficiaries see value in being able to see informal problem statements to which comments, updates, and trip reports could be attached. Our program can collect this data.
• Beneficiaries wish to share informal information sources such as news articles. Our program can collect this data.
• Beneficiaries want email notifications for Requests for Orders. Our program can collect this data.
• Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. Our program can collect this data.

Key Insights

From a product standpoint, much of this week was about ensuring the technical viability of the functions we’ve discussed up to this point and further confirming the value of collecting certain kinds of data with different beneficiaries. The trouble with collecting trip reports is well established and it seems AARs and SITREPs will require a similar system to collect, so we will know for sure if we are gathering those appropriately once we do a live test. O&M Logs and Travel Vouchers will be easy to collect and can be tied to system reminders to submit things like trip reports and AARs. RFOs are also easy to collect and the form could even be modified so that we can easily identify which trips apply to tech scouting.

From a deployment standpoint, we spoke with signal squadron about exactly how the process would work. They would first deploy it in a test environment with test users to understand the configurations and in the meantime set up a Change Advisory Board to determine the configuration for actual deployment. Due to the complexity of configurations on government systems, it’s unclear exactly how long such a process would take.

From a partnership standpoint, we spoke with a representative from Decisive Analytics, who saw the immediate value in our data gathering system working in tandem with their data processing system. We also opened up the prospect of speaking to other firms in the tech scouting world through our AFWERX contact. While such talks are preliminary, it appears that our concept is capable of stirring interest outside the context of our problem sponsor.

Interviews (10/102)

Brian – Signal Squadron

• Believes squadrons should be able to state or describe their problems in AARs.
• Sees value in CDD being able to look at all AARs and see how many people are complaining about the same capability gap. [noteworthy – he is the first person outside CDD to say this]
• Hopes that the program will enable a back and forth between groups that doesn’t require a ton of additional meetings or sifting through paperwork.

MAJ Tony – AFWERX

• Proposed connecting us with companies working in the tech scouting/tech adoption space that are not as sophisticated in the execution of software development.
• Acknowledged that security clearances could be a challenge. Did not think we could just get them on the possibility of the program being successful.
• Suggested that the best way forward is to make the MVP for unclassified activities. If our MVP is successful in an unclassified environment and could be useful for classified information, that can “prove” your need for clearance to take the project to the next level.

SGM Angel – CDD – Future Concepts

• Holds the general belief that squadrons should handle scouting for emerging tech that is currently available in industry.
• Was surprised to hear that some squadrons believed CDD should handle those kinds of activities. [worth noting that we did not use Angel’s distinction of tech that is currently available on the market when talking to squadrons]
• Thinks preferences may be personality dependent. But hopefully the unit reorganization will manage the difference in personalities – the Squadron rep and CDD rep will be side by side making decisions on who should go where under what circumstances.

Brian – Signal Squadron – Network Operations

• Trip reports are out there but not in one place that’s easy to find.
• RFO forms can be pulled. Will include dates, may be vague on trip details. We are trying to implement the use of a code to classify the kind of trip. Two categories in particular pertain to the tech scouting element. O&M log forms could also be pulled pretty easily.
• Travel Vouchers, AARs, and SITREPs are out there as well and we can get to them, but we kind of have to know what we’re looking for. Won’t just be able to pull from one place.
• We already have a database of vendors you can pull from as well.

MSG Dave – H/A Squadrons

• Does not believe CDD reorganization plan will solve the unit’s problems.
• Current job is to locate with A squadron to report their needs back to H squadron, and even though he’s there it’s a challenge to get in tune with what they are doing.
• Suspects that problem would repeat itself between CDD and the specialty squadrons.

MSG Joe – CDD – C4I

• Thinks challenges associated with RFO could be fixed with a vendor field and knows they already have the database.
• Sees our program, the Decisive Analytics program, and a new project management database for CDD as “the trifecta.”
• Eventually, you might be able to pull some of the data you need from that PM database.

Josh – Amazon R&D

• Works upstream from development teams.
• Amazon is highly decentralized – this comes with some inefficiencies but it could be more costly to take too much time thinking about a centraliezd policy.
• That said, his role is to risk assess a wide range of projects, then send the appropriately derisked ones down to R&D.
• Believes he is generally up to speed with the teams but the effort is active and informal. Holds tech “speed dating” meetings twice a year.

SGM Katie – CDD – Tech Targeting

• RFO descriptions are generally vague and not particularly informative.
• Does not think this is on purpose and could probably be changed to an extent, but unsure they could affect all elements in the building.
• Inevitably there will be some blind spots on the description itself, but knowledge of the trip itself and some of the other details on the form can be valuable.

Mark – Decisive Analytics

• Already worked with several other DoD agencies, achieved necessary clearances.
• Liked the idea of bouncing some of the data we would collect, particularly trip reports, off web services.
• Suggested there were other ways of working together if this particular project did not work out.

Chris – Signal Squadron – Network Operations

• To deploy a piece of software like yours, we would have a team of engineers deploy it in a test environment first to figure out configurations.
• Then we can bring in users to test it, push it through the Change Advisory Board (implementation) process while that is happening.
• Government networks have a lot of configuration, CAB is just there to make sure it all happens functionally.
• Should be workable to allow access based on roles within unit and clearances.

Hypotheses Confirmed/Debunked

• Confirmed: Beneficiaries see value in being able to see informal problem statements to which comments, updates, and trip reports could be attached. Interviewees have consistently expressed interest in being able to see problem statements. Partially confirmed: Our program can collect this data. Beneficiaries suggested such data could be pulled from the After Action Report process, though these forms suffer from the same inconsistency challenges of trip reports.
• Partially Confirmed: Beneficiaries wish to share informal information sources such as news articles. Beneficiaries do not consistently express interest in this but some are highly interested. Partially Confirmed: Our program can collect this data. It could be challenging for our program to incorporate this kind of information by itself but Decisive Analytics seems to have a better handle on the elements that make this data challenging to collect. As such, it’s more within the realm of possibility.
• Debunked: Beneficiaries want email notifications for Requests for Orders. There are certain kinds of info on RFOs that could be valuable and help to make building out trip reports more easy. As such, beneficiaries see them as helpful within the system. However, they do not want to look at them directly as they’re not informative enough on their own. Confirmed: Our program can collect this data. Not only are RFOs readily available for collection, we could slightly alter the existing form to easily categorize trips relevant to our system and set up reminders to submit trip reports.
• Debunked: Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. Beneficiaries again identified some information on the forms that could be useful, but would not want to look them up directly. Confirmed: Our program can collect this data. The database itself is readily available. If the information on the forms were to be configured appropriately it could yield some helpful information. However, given the general lack of enthusiasm it is low priority.

Questions for Next Week

What would a test run of the program look like outside USASOC?

Who among our existing contacts can help us with another kind of test run?

What does USASOC need from us to have a more complete timeline on deployment?

To what extent will we need to be involved with USASOC once we reach the deployment stage?

Questions for this Week

Is our input system more convenient than whatever people are doing now?

What amount of limited information is even the most paranoid R&D employee willing to share freely?

How can the search interface be condensed effectively? What kind of info should a condensed version display?

What value can CDD deliver that makes communicating with them worth the risk of getting a project shut down?

What use cases do our beneficiaries find most valuable? How many forms could our input system replace? Which forms do beneficiaries see direct value in searching?

What We are Testing

• Beneficiaries see value in searching and subscribing to After Action Reports.
• Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor.
• Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
• Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it.
• Beneficiaries wish to share informal information sources such as news articles.
• Beneficiaries want email notifications for relevant trip reports.
• Beneficiaries want email notifications for Requests for Orders.
• Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems.
• R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD.

Key Insights

This week torpedoed some of our assumptions regarding how much information we could glean from certain forms and how much value our beneficiaries saw in accessing them. But it also torpedoed some of our assumptions about how much people would be willing to participate in our system. Request for Orders forms, O&M log forms, and even requirements all met some skepticism when we described their case uses on our system. In each case, we will need to determine if we can consistently pull specific types of information from those forms because we can’t rely on people digging deeper the way we would with trip reports.

However, we were also surprised to see the level of interest in having the ability to submit informal pre-requirement problem statements. We were also relieved to find out that R&D’s concerns about CDD interference would not need to be completely eliminated in order to submit their data. Indeed, our program can even be framed as shielding R&D from certain kinds of unwanted contacts because CDD employees can add all of their updates on a given capability to the central database and R&D can choose to inquire about the things they find relevant to their problems. We must remain cognizant of this relationship and other ways that problems could manifest, but it does not appear to be as serious an obstacle as we once anticipated.

Interviews (10/92)

MSG Matt – C4I

• [examined the wire framing of our mockups]
• We need to be able to at least save as a link any kind of attached documentation. Ideally it’s drag and drop and gets stored somewhere else so it doesn’t slow down the search, you only go for that extra info if you want to deep dive.
• Thinks we may need a reminder system to get people to input stuff on their trips to “capability providers” could be as simple as checking a box on a Request for Orders form that we’d be collecting data from anyway.
• We are hoping at some point in the future to integrate this system with another we currently own the IP on that would let us use this data to populate a sort of capability Wiki.

MSG Joe – C4I

• [examined the wire framing of our mockups]
• My issue with the trip report submission page is that one comment box is too unstructured. I know it’s an informal process that went by the wayside, but I think this is an opportunity to at least loosely organize them into a few fields.
• Even something like “Who, What, Where, When, Why, How?” fields could be immensely helpful.
• Likes that tags eliminate the need for quite a few fields and enables splitting up the trip report a bit.
• Regarding Request for Orders forms prompting a trip report: it’ll help the commodity area people give a heads up on what a capability provider is like prior to the meeting. But it’s also insurance for the R&D person: if they posted that they were having this meeting and nobody from CDD follows up, the R&D person can claim they did their part.

MSG[?] Sandy – C4I

• [examined wire framing of our mockups]
• Small procedural point – sometimes trip dates change, so if you don’t ask after the fact then the dates might be wrong.
• Ideally if I don’t know who to talk to in a commodity area, I should be able to identify the commodity area itself and know that the right person in that commodity area will be notified.
• Category-wise, all of our capability gaps are getting funneled through our commodity areas one way or another, so at least getting the right capabilities into their commodity area buckets is very helpful.

Nic – Squadron R&D Deployment [Search MVP Test]

• Would want to be able to subscribe to keywords related to his commodity area and receive email updates on trip reports, but could see his inbox getting clogged up.
• Wants to be able to subscribe to certain requirements and automatically get updates on trip reports related to those.

MAJ Josh – R&D Section Leader

• Generally has a positive relationship with CDD. Sees them as primarily focused on R&D, while squadrons are also focused on operations.
• CDD makes life harder when they’re not in sync with what the operational guys want, or if we have to chase them down in those times where we do want their help.
• The incentive and disincentive against sharing is money – sharing can open you up to new sources of funding but it can also open you up to scrutiny and competing projects.
• Sees no downside to submitting a preliminary description of what he is working on before he really gets started – the disincentive against sharing in his opinion comes once time and money is already invested.
• His team needs status updates on ongoing procurement efforts, work on new procurement efforts, monitoring of emerging tech, and summary reports from trips.

Brian – Signal Squadron

• Confirmed interest in seeing any After Action Reports related to integrating new tech into systems he is working on.
• Confirmed interest in searching by a specific vendor and looking up all trips related to that vendor.
• Confirmed interest in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
• Confirmed interest in subscribing to specific existing requirements and getting updates about trips related to it.
• Confirmed interest in sharing articles related to a particular topic within the system.

Doug Brook

• In his experience, the problem wasn’t so much that people were never willing to change. It was that once they decided they didn’t want to change, it was impossible to move forward.
• Recalled a major overhaul to DoD’s personnel system. Army and Air Force were on board because they didn’t care much for their systems. Navy did not want to change, and Marines were particularly opposed because they had just overhauled their own system. The overhaul subsequently never really got off the ground anywhere.
• Did not recall a single time when he was able to force a stubborn organization to implement a policy. Always found the best way was to work with those boundaries as best as possible.
• Agreed with a bottom up approach – only working with the organizations that express a need for what we are doing helps reduce the chance of someone sinking the program, likewise for focusing on buy in from end users.
• Agreed that if our premise was correct that most organizations didn’t really have a good system for what we were doing it would be a real advantage in avoiding institutional resistance.

Erik Stayton – User Experience Researcher, Nissan

• Skeptical of people who claim to hate they computers in the sense that he doubts they specifically hate computers. Rather, they’ve never found them useful in doing their jobs. If they don’t see the value there, they’ll be reluctant to change, even if the current method is imperfect.
• So part of the effort will be to get people to stop worrying about the database, but part will also be changing what database means to them.
• They could also be afraid of the implied change of a new system, and might have some additional fears you’re not picking up on yet. But to some extent you can’t totally control this. If it’s a cultural problem, there’s going to need to be change regardless.

Nic – Squadron R&D Deployment [Input MVP Test]

• Would use the current tagging system to file his trip reports.
• Would also find value in being able to submit “pre-requirement” problem statements to get notified if a similar problem statement already exists and whether or not CDD is working on it.

SGM Katie

• Identified searching for external vendors to see who contacted them previously and tagging trip reports to notify any interested parties as high priority.
• Would want email notifications for relevant trip reports but not Requests for Orders – thinks there is too much variance in RFOs to get useful information out of them.
• Does not believe that log forms will help them identify “inappropriate” O&M purchases – there simply isn’t enough transparency with those purchases to identify remotely. It could help on a smaller scale within R&D sections though.
• Would also want to be able to subscribe to requirements, but doesn’t believe it will be as high priority as subscribing to and tagging vendors.
• Believes submitting informal problem statements would help as long as people self-identify when submitting so CDD can track them down and make use of such info.

Hypotheses Confirmed/Debunked

• Confirmed: Beneficiaries see value in searching and subscribing to After Action Reports. Our CDD beneficiaries see value in checking back to see which tech is actually being used, R&D beneficiaries often work on systems where other techs get integrated and would want to know of their existence.
• Confirmed: Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor. Different beneficiaries use this information for different purposes but consistently identify finding the appropriate trip reports as the best way of getting the information they need.
• Partially Confirmed: Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached. Several beneficiaries said this would be valuable information but did not necessarily confirm that they would be interested in submitting their own problem statements.
• Confirmed: Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it. Beneficiaries continue to look for an effective replacement for their current requirement tracker but don’t see requirements as high priority compared to the trip reports. However, the email notification system is perfect for this particular area, as they’re also specific enough that there was less concern about getting spammed.
• Partially Confirmed: Beneficiaries wish to share informal information sources such as news articles. Playing off some people’s suggestions that this kind of functionality be included, we asked some others who had made no mention of informal sharing in the past and they confirmed it would be helpful but were not particularly excited about it.
• Partially Confirmed: Beneficiaries want email notifications for relevant trip reports. However, they want some ability to be selective to avoid the spam problem mentioned above.
• Partially Debunked: Beneficiaries want email notifications for Requests for Orders. Nobody expressed a direct interest in seeing the forms. We have reason to believe that certain information that seems to show up consistently on RFOs could help but will have to reframe as nobody is looking at the forms themselves.
• Mostly Debunked: Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. People saw use for this in some of the tighter knit circles but for the most part didn’t think they could glean anything useful from the data that’s available on a Log Form.
• Confirmed: R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD. Beneficiaries clarified that the risk of friction between R&D and CDD exists in to main forms: when R&D has already invested time and money in a project and when CDD imposes certain ideas on R&D’s busy schedule. The former could potentially be addressed if CDD responds to informal updates quickly enough. The latter could be addressed through the system – CDD can put their findings in the database and R&D can take or leave what they find useful based on their specific context.

Questions for Next Week

Do the forms that beneficiaries expressed little interest in offer any kind of data that we could use consistently in our system?

How could informal problem statements work without overcomplicating the existing system?

Would CDD agree to use our system in a way that we think mitigates R&D concerns? Is it different from how they intended to use it?

Do any end users of capabilities outside R&D intend to use our program?

Can our existing functions be leveraged to accomplish some of the other goals mentioned in the original problem statement?

What will a proper, real world test of our system look like?