Questions for this Week
Is our input system more convenient than whatever people are doing now?
What amount of limited information is even the most paranoid R&D employee willing to share freely?
How can the search interface be condensed effectively? What kind of info should a condensed version display?
What value can CDD deliver that makes communicating with them worth the risk of getting a project shut down?
What use cases do our beneficiaries find most valuable? How many forms could our input system replace? Which forms do beneficiaries see direct value in searching?
What We are Testing
- Beneficiaries see value in searching and subscribing to After Action Reports.
- Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor.
- Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
- Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it.
- Beneficiaries wish to share informal information sources such as news articles.
- Beneficiaries want email notifications for relevant trip reports.
- Beneficiaries want email notifications for Requests for Orders.
- Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems.
- R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD.
This week torpedoed some of our assumptions regarding how much information we could glean from certain forms and how much value our beneficiaries saw in accessing them. But it also torpedoed some of our assumptions about how much people would be willing to participate in our system. Request for Orders forms, O&M log forms, and even requirements all met some skepticism when we described their case uses on our system. In each case, we will need to determine if we can consistently pull specific types of information from those forms because we can’t rely on people digging deeper the way we would with trip reports.
However, we were also surprised to see the level of interest in having the ability to submit informal pre-requirement problem statements. We were also relieved to find out that R&D’s concerns about CDD interference would not need to be completely eliminated in order to submit their data. Indeed, our program can even be framed as shielding R&D from certain kinds of unwanted contacts because CDD employees can add all of their updates on a given capability to the central database and R&D can choose to inquire about the things they find relevant to their problems. We must remain cognizant of this relationship and other ways that problems could manifest, but it does not appear to be as serious an obstacle as we once anticipated.
MSG Matt – C4I
- [examined the wire framing of our mockups]
- We need to be able to at least save as a link any kind of attached documentation. Ideally it’s drag and drop and gets stored somewhere else so it doesn’t slow down the search, you only go for that extra info if you want to deep dive.
- Thinks we may need a reminder system to get people to input stuff on their trips to “capability providers” could be as simple as checking a box on a Request for Orders form that we’d be collecting data from anyway.
- We are hoping at some point in the future to integrate this system with another we currently own the IP on that would let us use this data to populate a sort of capability Wiki.
MSG Joe – C4I
- [examined the wire framing of our mockups]
- My issue with the trip report submission page is that one comment box is too unstructured. I know it’s an informal process that went by the wayside, but I think this is an opportunity to at least loosely organize them into a few fields.
- Even something like “Who, What, Where, When, Why, How?” fields could be immensely helpful.
- Likes that tags eliminate the need for quite a few fields and enables splitting up the trip report a bit.
- Regarding Request for Orders forms prompting a trip report: it’ll help the commodity area people give a heads up on what a capability provider is like prior to the meeting. But it’s also insurance for the R&D person: if they posted that they were having this meeting and nobody from CDD follows up, the R&D person can claim they did their part.
MSG[?] Sandy – C4I
- [examined wire framing of our mockups]
- Small procedural point – sometimes trip dates change, so if you don’t ask after the fact then the dates might be wrong.
- Ideally if I don’t know who to talk to in a commodity area, I should be able to identify the commodity area itself and know that the right person in that commodity area will be notified.
- Category-wise, all of our capability gaps are getting funneled through our commodity areas one way or another, so at least getting the right capabilities into their commodity area buckets is very helpful.
Nic – Squadron R&D Deployment [Search MVP Test]
- Would want to be able to subscribe to keywords related to his commodity area and receive email updates on trip reports, but could see his inbox getting clogged up.
- Wants to be able to subscribe to certain requirements and automatically get updates on trip reports related to those.
MAJ Josh – R&D Section Leader
- Generally has a positive relationship with CDD. Sees them as primarily focused on R&D, while squadrons are also focused on operations.
- CDD makes life harder when they’re not in sync with what the operational guys want, or if we have to chase them down in those times where we do want their help.
- The incentive and disincentive against sharing is money – sharing can open you up to new sources of funding but it can also open you up to scrutiny and competing projects.
- Sees no downside to submitting a preliminary description of what he is working on before he really gets started – the disincentive against sharing in his opinion comes once time and money is already invested.
- His team needs status updates on ongoing procurement efforts, work on new procurement efforts, monitoring of emerging tech, and summary reports from trips.
Brian – Signal Squadron
- Confirmed interest in seeing any After Action Reports related to integrating new tech into systems he is working on.
- Confirmed interest in searching by a specific vendor and looking up all trips related to that vendor.
- Confirmed interest in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached.
- Confirmed interest in subscribing to specific existing requirements and getting updates about trips related to it.
- Confirmed interest in sharing articles related to a particular topic within the system.
- In his experience, the problem wasn’t so much that people were never willing to change. It was that once they decided they didn’t want to change, it was impossible to move forward.
- Recalled a major overhaul to DoD’s personnel system. Army and Air Force were on board because they didn’t care much for their systems. Navy did not want to change, and Marines were particularly opposed because they had just overhauled their own system. The overhaul subsequently never really got off the ground anywhere.
- Did not recall a single time when he was able to force a stubborn organization to implement a policy. Always found the best way was to work with those boundaries as best as possible.
- Agreed with a bottom up approach – only working with the organizations that express a need for what we are doing helps reduce the chance of someone sinking the program, likewise for focusing on buy in from end users.
- Agreed that if our premise was correct that most organizations didn’t really have a good system for what we were doing it would be a real advantage in avoiding institutional resistance.
Erik Stayton – User Experience Researcher, Nissan
- Skeptical of people who claim to hate they computers in the sense that he doubts they specifically hate computers. Rather, they’ve never found them useful in doing their jobs. If they don’t see the value there, they’ll be reluctant to change, even if the current method is imperfect.
- So part of the effort will be to get people to stop worrying about the database, but part will also be changing what database means to them.
- They could also be afraid of the implied change of a new system, and might have some additional fears you’re not picking up on yet. But to some extent you can’t totally control this. If it’s a cultural problem, there’s going to need to be change regardless.
Nic – Squadron R&D Deployment [Input MVP Test]
- Would use the current tagging system to file his trip reports.
- Would also find value in being able to submit “pre-requirement” problem statements to get notified if a similar problem statement already exists and whether or not CDD is working on it.
- Identified searching for external vendors to see who contacted them previously and tagging trip reports to notify any interested parties as high priority.
- Would want email notifications for relevant trip reports but not Requests for Orders – thinks there is too much variance in RFOs to get useful information out of them.
- Does not believe that log forms will help them identify “inappropriate” O&M purchases – there simply isn’t enough transparency with those purchases to identify remotely. It could help on a smaller scale within R&D sections though.
- Would also want to be able to subscribe to requirements, but doesn’t believe it will be as high priority as subscribing to and tagging vendors.
- Believes submitting informal problem statements would help as long as people self-identify when submitting so CDD can track them down and make use of such info.
- Confirmed: Beneficiaries see value in searching and subscribing to After Action Reports. Our CDD beneficiaries see value in checking back to see which tech is actually being used, R&D beneficiaries often work on systems where other techs get integrated and would want to know of their existence.
- Confirmed: Beneficiaries see value in searching by a specific vendor and looking up all trips related to that vendor. Different beneficiaries use this information for different purposes but consistently identify finding the appropriate trip reports as the best way of getting the information they need.
- Partially Confirmed: Beneficiaries see value in being able to submit an informal problem statement to which comments, updates, and trip reports could be attached. Several beneficiaries said this would be valuable information but did not necessarily confirm that they would be interested in submitting their own problem statements.
- Confirmed: Beneficiaries see value in subscribing to existing requirements and getting updates about trips related to it. Beneficiaries continue to look for an effective replacement for their current requirement tracker but don’t see requirements as high priority compared to the trip reports. However, the email notification system is perfect for this particular area, as they’re also specific enough that there was less concern about getting spammed.
- Partially Confirmed: Beneficiaries wish to share informal information sources such as news articles. Playing off some people’s suggestions that this kind of functionality be included, we asked some others who had made no mention of informal sharing in the past and they confirmed it would be helpful but were not particularly excited about it.
- Partially Confirmed: Beneficiaries want email notifications for relevant trip reports. However, they want some ability to be selective to avoid the spam problem mentioned above.
- Partially Debunked: Beneficiaries want email notifications for Requests for Orders. Nobody expressed a direct interest in seeing the forms. We have reason to believe that certain information that seems to show up consistently on RFOs could help but will have to reframe as nobody is looking at the forms themselves.
- Mostly Debunked: Beneficiaries want to look up O&M Log Form purchases to identify potential procurement problems. People saw use for this in some of the tighter knit circles but for the most part didn’t think they could glean anything useful from the data that’s available on a Log Form.
- Confirmed: R&D users have some willingness to share info despite the apparent risks posed by interacting with CDD. Beneficiaries clarified that the risk of friction between R&D and CDD exists in to main forms: when R&D has already invested time and money in a project and when CDD imposes certain ideas on R&D’s busy schedule. The former could potentially be addressed if CDD responds to informal updates quickly enough. The latter could be addressed through the system – CDD can put their findings in the database and R&D can take or leave what they find useful based on their specific context.
Questions for Next Week
Do the forms that beneficiaries expressed little interest in offer any kind of data that we could use consistently in our system?
How could informal problem statements work without overcomplicating the existing system?
Would CDD agree to use our system in a way that we think mitigates R&D concerns? Is it different from how they intended to use it?
Do any end users of capabilities outside R&D intend to use our program?
Can our existing functions be leveraged to accomplish some of the other goals mentioned in the original problem statement?
What will a proper, real world test of our system look like?