Category Archives: Project

Project Presentations

On Tuesday, 12/6, or Thursday, 12/8.

General Directions

The project presentation is intended to provide a high-level overview of your project to an audience of your peers and anyone knowledgeable in CS. CS faculty and anyone interested in CSEd will be invited to see your presentation.

The presentation should demonstrate your ability to communicate the significance and interpret the findings of your research project. The presentation should stand on its own so that it makes sense to someone who has not read any of your other work.

Each group will have a 20-minute slot. 10-15 minutes should be the presentation and the rest of the time is for questions. Everyone in the group should have a turn to present. The presentation is serving as a summative assessment to confirm each of you can comprehensively explain some aspect of your group’s work. If someone cannot make the presentation, we will Zoom them in so they can participate in the Q/A. If that is not possible, they should prepare a recorded video of their part of the presentation. I strongly recommend you practice as a group to get a sense of the timing.

Presentation Content

I recommend starting with the order and content in your report. And then, set aside your report and talk your way through the presentation, reordering slides and points until it more naturally flows when explaining your project to an interested listener. Your presentation should include all of the following, but does not need to be in this order:

  1. Introduction: Motivate why your research question is interesting and introduce your research questions
  2. Related work: This does not need to be as in-depth as your report. It should, at minimum, cover anything that is important to know to understand the rest of your presentation.
  3. Methods and Results: Organize this similarly to how I suggested you organize this section in your report. In addition, remind your audience of research questions as you answer them.
  4. Limitations, discussion, and future work
  5. Summary slide: This is a combination of conclusions and any other information that might be helpful to the audience. This is your last slide. There is no need to have a slide that just says “Questions.” Ending on this slide will help your audience remember your presentation and prompt them with things they may want to ask a question about.

Grading Rubric

The presentation will be graded as the following pieces, each with our usual four-step rubric scale as follows.

Presentation slides (10 points)

  • Exemplary (10 points) – The slides have all of the sections, are well organized, and are reasonably easy to follow and read.
  • Satisfactory (9 points) – One or two sections are not quite sufficiently filled out OR some of the slides are hard to follow or read.
  • Not yet (6 points) – More than 2 sections are missing, the slides are disorganized, and the presentation was hard to follow.
  • Unassessable (2 points) – The presentation exists, but it is severely lacking.

Presentation by each person (10 points)

  • Exemplary (10 points) – The presenter presented at least one section, clearly showed a mastery of the material they presented, and they were reasonably understandable in their explanation.
  • Satisfactory (9 points) – The presenter equally contributed to the slide deck and did everything in their power to present either in person, remotely, or via video, but life got in the way. OR the presenter presented their section with a little lack of mastery, mainly evident by having trouble answering a question.
  • Not yet (6 points) – The presenter equally contributed to the slide deck but clearly did not practice the slides, evidenced by substantial pauses, stumbling over points, or saying incorrect information.
  • Unassessable (2 points) – The presenter helped create a slide in the deck but otherwise did not contribute.

All members of a team may not necessarily receive the same grade.

Project Final Report

Due: Friday 12/9, late due Sunday 12/11

General Directions

Your final report is a comprehensive account of your project and as if you were planning to submit it to a conference (without worrying about formatting). It should be written as if you had “planned this as your project all along.” A report is not a chronological story of your project. It is a summary of what you did where the “story” serves the reader’s comprehension. Just like all of the related work you have read is framed not as a chronology but as a summary of what was done and found.

The report should stand on its own so that it makes sense to someone who has not read your proposal or prototype. It should be 5-7 pages (not counting references) using standard margins (1 in.), font (11-12 pt), and line spacing (1-1.5) OR you can use the ACM standard 2-column template. A typical submission is around 3-4 pages of text and 5-7 pages overall with tables and figures. Your report should have a title and your names with netids.

For citations, use the same notation that is common in the ACM papers (SIGCSE, ICER, ITiCSE, etc.) and cite the work by saying something like “In Smith et al.’s [3] work, ….” Note the use of the [#] as an annotation as opposed to a noun “In [3], …”

You may also include an appendix with as many pages as you need. This should mainly be of tables and figures, not text. The only text that really belongs in the appendix is any captions that help explain tables and figures. However, the report should be understandable without the appendix. The appendix is just a place for supplemental and extra information. So when in doubt, if the reader needs something, just put it in the appendix unless you find yourself referencing it a lot.

You should convert your written report to a pdf and upload it to Gradescope under the assignment “Project Final Report” by the due date. Use the group submission feature on Gradescope. You do not need to upload your accompanying data, code, or other supplemental resources demonstrating your work to Gradescope; instead, your report should contain instructions on how to access these resources (see part 2 below for more details).

Checklist for this section

  1. 5-7 pages (not counting references)
  2. Standard margins, spacing, and font
  3. Citation style uses “In first_author’s_last_name et al. [#], …”

General Feedback from Prototype

  • Provide more context about your data. While Prof. Stephens-Martinez knows where you got the data, at this point, you know more about it than she does. Also, your classmates (to who you will be presenting) are not familiar with your data set. Therefore near the beginning of your methods/results section should be a section about the data that answers the following:
    • Where is your data from? (ex: “We collected data from the website https://www.pokemon.com/us/pokedex/ to create a database of pokemon.”)
    • When was it collected? (ex: “We collected all pokemon that existed from 1996 – 2016.”)
    • What is the demographic breakdown as is relevant to your question? (ex: “Table A shows the breakdown of pokemon in our data set by generation and their primary type for the 5 most common types [which are the types that are relevant to our research question].”)
    • And any other information that helps contextualize your data to help the reader understand how to interpret it.

Part 1: Introduction and Research Questions

Your final report should begin by motivating your topic and stating your research question(s). In contrast to the prior reports, the final report does not need to explicitly justify that the research questions are substantial and feasible in the text; your results should demonstrate both of these points.

You can start with the text from your prototype, but you should update your introduction and research questions to reflect changes in or refinements of the project vision. And there should not be a section comparing this report with prior ones. Remember, this report is as if you had “always planned” to do what you did and were submitting it to a conference. Your introduction should be sufficient to provide context for the rest of your report.

Checklist for this section

  1. Introduces topic
  2. Motivates research question
  3. Defines one or more research questions – Exemplary would clearly label these, such as having them be in a numbered list
  4. Includes citations as needed for an introduction unless it’s clear the introduction does not need any citations

Part 2: Related Work

This section should summarize the work you found that is related to your project. It should be organized by the big ideas and summarize the key takeaways generally with supporting citations. Remember the “how to write briskly” reading and that you can always use another paper as an example of how to write your own related work section.

Checklist for this section

  1. Organized by the big ideas and is a coherent whole
  2. Summarizes the key takeaways for all related work mentioned that are relevant to this work
  3. Includes citations – Remember, the citation style mentioned in the general directions

Part 3: Methods and Results

This section should summarize how you answered your research questions and the results of that analysis. Often there are two “sets” of methods, the methods used to answer most/all of the research questions, like cleaning and transforming the data, and the methods used for a particular research question. The former should be in its own section, so the text is not repetitive. The latter is in its respective results section for the sake of proximity.

Your report should be specific about exactly what data were used and how the results were generated. For example, if you filtered out some of the data due to A and B reasons, you should state what criteria were used to filter the data, why, and how much of the data was filtered out (or is left). These steps should be explained in enough detail such that an informed reader (like another group working on the same data set) could reasonably be expected to reproduce your results with time and effort. Just saying, for example, “we cleaned the data and dealt with missing values” is not sufficient detail.

Results should be summarized using clearly labeled tables or figures and supplemented with written explanations of the significance of the results with respect to the research questions outlined previously.

Your report should also contain instructions on how to access your full implementation (that is, your code, data, and any other supplemental resources like additional charts or tables). The simplest way to do so is to include a link to the box folder, GitLab repo (share it with Prof. Stephens-Martinez’s netid email: kvs13@duke.edu), or whatever other platforms your group is using to house your data and code. Remember to keep the data private!

Checklist for section

  1. There is enough information about the context of your data to understand your results – See the general feedback above about providing context about your data.
  2. Summarizes the methods used to transform the data into what was used to answer each research question – If the data you had was readily available with no transformation, this subsection should simply say where the data came from.
  3. Explains the methods used to answer each research question
  4. Reports the results of each research question

Part 4: Limitations, Discussion, and Future Work

In this part, you should discuss any important limitations or caveats to your results with respect to answering your research questions. For example, if you don’t have as much data as you would like or are unable to fairly evaluate the performance of a predictive model, explain and contextualize those limitations.

Besides limitations, put any other discussions or ideas for future work here. This could be a discussion on an idea that explains the results you had, but you do not have the data to provide evidence for the idea. This could explain how future research might address the limitations you outline, or it could pose additional follow-up research questions based on your results so far. In short, explain how an informed reader (such as a peer in the class) could improve on and extend your results.

Checklist for section

  1. Outlines limitations or caveats to the work
  2. Discusses the works results or future work

Part 5: Conclusion

This section should provide the key takeaways from your work and should only be a few paragraphs at most.

Checklist for section

  1. The section is a few paragraphs at most
  2. Fully addresses all research questions stated in the introduction

(Optional) Part 6: Appendix of Additional Figures and Tables

If you are struggling to keep your report within the 5-7 page limit, you may move some of your figures and tables to an optional appendix that will not count against your page limit. However, your report should stand on its own without the appendix. The appendix is for adding more nuance to your results, not to give you more space to talk about your results. Succinctness is an important skill to practice. Your paper will be graded without looking at the appendix.

Feedback and Grading Rubric

Each section will be graded on a four-step rubric scale as follows.

  • E (Exemplary) – Work that meets all requirements of that section.
  • S (Satisfactory) – Work that meets all requirements with only slight mistakes or missing pieces of information.
  • N (Not yet) – Work that does not meet some requirements and/or displays developing or incomplete work that needs substantial revision to meet satisfactory standards.
  • U (Unassessable) – Work that is missing, does not demonstrate meaningful effort, or does not provide enough evidence to determine a level of mastery.

The entire assignment is worth 100 points.

  • 10 points will be allocated for meeting general directions (length, on-time pdf submission, group submission, etc.). You cannot submit a proposal greater than 3 pages. Learning how to be succinct is an important skill.
  • 18 points are allocated for each section. (18*5 = 90)

The rubric will be converted to points as follows:

  • E = full credit
  • S = E_full_credit – 1
  • N = E_full_credit / 2
  • U = E_full_credit / 5
  • Blank = 0

Anything earning less than an E will receive feedback in Gradescope. If your proposal earns less than an S in any section, you will be allowed 2 resubmissions to bring it up to the E or S standards for all sections. If your report earns E’s and S’s only, you can have 1 resubmission if your group decides to aim for a higher score. I will aim to give you feedback by Tuesday, 12/13, so a resubmission must be given by Friday, 12/16.

Writing workshop: Project Abstract and more

For this class, we will start with a presentation by Shao-Heng. He will be practicing his presentation for his SIGCSE TS 2023 paper “What Drives Students to Office Hours: Individual Differences and Similarities.” This presentation will serve as an example of what a polished Exemplary talk looks like to give everyone an idea of what kind of talk to strive for the following week. Note that Shao-Heng’s talk is polished by getting a lot of feedback from me and will not be used as the floor of the Exemplary standard but as something to aim for.

Then, we will have another writing workshop with the goal of writing an abstract about your project. These abstracts will be included in Prof. Stephens-Martinez’s advertisement to the CSEd faculty in the department, inviting them to come to see your presentations. Besides getting feedback on your abstract, you can get feedback on anything else, including your presentation next week.

Your abstract is due Thursday, 12/1 (the day of this class), at 11:59 PM. Your abstract should be 1-3 paragraphs and no more than half a page long. Formatting is not relevant since you will submit via an open textbox in Gradescope.

At this point, you’ve read many abstracts, so you should feel free to draw on what you’ve liked and not liked about those abstracts. The following should be included in your abstract:

  1. Why is it important? (1-2 sentences)
  2. What you did (2-4 sentences, e.g., “We investigated <what your data is> to answer <your research questions>”)
  3. What you found (2-5 sentences, e.g., “We found….”)
  4. Concluding sentence (1-2 sentences, optional)

This will count as 1% of the project presentation’s 12% for your overall grade.

Grading

  • Exemplary (10 points) – The abstract is well-written and has all of the required pieces.
  • Satisfactory (9 points) – The abstract has all of the required pieces, but some of it is confusing or vague.
  • Not yet (6 points) – The abstract is clearly missing one or more parts.
  • Unassessable (2 points) – There is an abstract but it does not fulfill the Not yet criteria.

Group Check-ins and Learning Theory in CS

We will spend most of the class period doing one-on-one group check-ins. You will have feedback from your prototype by this day, so you can use the time to go over this feedback. When not meeting with Prof. Stephens-Martinez, plan to work or meet with your group, like a writing workshop.

Make sure to fill out a group check-in slide!

In the last 30 minutes, we will have Prof. Brandon Fain visit again to continue our conversation from last Thursday. To prepare for the discussion, answer the following questions in your QQC Docs the day before, Wednesday 11/16.

  1. Thinking about pedagogy (apart from curriculum and motivation), what has been particularly effective and/or ineffective in helping you to learn computing theory?
  2. What is one idea from the reading that stuck out to you, and how could it be applied?

Related work writing workshop & Group Check-ins

For this class period, we will sit with our teams and start with an “I’m looking for a paper on….” exercise. We’ll do it on the Ed post for this class day, and you can feel free to start adding paper requests now. Then we’ll spend the rest of the time silently working while Prof. Stephens-Martinez check-ins with each team quietly to give feedback on their related work section and do a group check-in.

You need to prepare the following:

  1. A related work document your team wants feedback on.
  2. A group check-in slide in the group check-in google slide deck (link in the Ed post for this class day).

Project Prototype

Due: Thursday, 11/10

General Directions

The prototype deliverable is intended to demonstrate a proof of concept for your final project report. Large multi-week projects are challenging, this deliverable is intended to provide additional structure to ensure you are making progress and on a path toward success. It also is a good milestone to reevaluate if your current research question is a good course of action.

It consists of a written report detailed below, along with any accompanying data, code, or other supplementary resources that demonstrate your progress so far in the project. You can think of it as a rough draft for your final project. The report should stand on its own so that it makes sense to someone who has not read your proposal.

The report should contain at least three parts, which we define below. In terms of length, it should be about 3-4 pages using standard margins (1 in.), font (11-12 pt), and line spacing (1-1.5) OR you can use the ACM standard 2-column template (see general proposal feedback below). The page limit does not include your references. A typical submission is around 2-3 pages of text and 3-4 pages overall with tables and figures. You should convert your written report to a pdf and upload it to Gradescope under the assignment “Project Prototype” by the due date. Be sure to include your names and NetIDs in your final document and use the group submission feature on Gradescope. You do not need to upload your accompanying data, code, or other supplemental resources demonstrating your work to Gradescope; instead, your report should contain instructions on how to access these resources (see part 2 below for more details).

Checklist for this section

  1. 3-4 pages (not counting references)
  2. Standard margins, spacing, and font

General Feedback from Proposal

Something to keep in mind is the general feedback given about the proposal:

  • To cite a paper, consider using the same notation that is common in the ACM papers (SIGCSE, ICER, ITiCSE, etc.) and cite the work by saying something like “In Smith et al.’s [3] work, ….”
    • I (Prof. Stephens-Martinez) do not recommend using [#] as a noun (i.e. “[4] showed that…”), it is much harder to remember what a citation is about without at least the extra scaffolding information of who the first author was.
    • You could use the ACM template with two columns. If you want to use the LaTeX template, I’d be happy to own the Overleaf document if you need/want all of Overleaf’s features.
  • Many of the research questions were vague or large. Which is not surprising for the proposal. Going forward, I encourage you all to consistently go back to your research question and consider how to refine it to something more precise/smaller.
    • If your research question is not helping you make decisions about how to analyze something, that means it is too vague/large and needs to be refined enough that you can use it to help you make decisions.

Part 1: Introduction and Research Questions

Your prototype report should still begin by introducing your topic and stating your research question(s) as in your proposal. Your research question(s) should be substantial and feasible. Briefly justify each of these points as in the project proposal. You can start with the text from your proposal, but you should update your introduction and research questions to reflect changes in or refinements of the project vision. Make sure to include a subsection pointing out what has changed since the proposal. Your introduction should be sufficient to provide context for the rest of your report (a.k.a. your proposal should not be required reading to understand your report). You should start including citations in your introduction for statements that require a citation.

Checklist for this section

  1. Introduces topic
  2. Motivates research question
  3. Defines one or more research questions – Exemplary would clearly label these such as having them be a numbered list
  4. Description of what has changed, if things have changed – Exemplary has this easy to find
  5. Research questions are substantial and feasible
  6. Include at least some citations as needed for an introduction unless it’s clear the introduction does not need any citations

Part 2: Related Work

At this point in time, you should have read much of the related work you found for your proposal and likely found a few more. This section should now summarize the key takeaways of all of the papers you’ve found so far in some coherent whole. Remember the “how to write briskly” reading and that you can always use another paper as an example on how to write your own related work section.

If your related work is not done, this section should end with a subsection on what related work still needs to be found, your plan for finding it, and any questions you have for me when I give you feedback on this prototype.

Checklist for section

  1. Summarizes key takeaways of all papers read so far
  2. Includes citations for all work
  3. Section is a coherent whole
  4. If applicable, includes a section on what still needs to be found, a plan, and any questions/requests for feedback

Part 3: Preliminary Results and Methods

The preliminary results section of your report should summarize the results obtained so far in the project. Where possible, results should be summarized using clearly labeled tables or figures and supplemented with a written explanation of the significance of the results with respect to the research questions outlined in the previous section. Your results do not need to be final or conclusive for your entire project but should demonstrate substantial effort and progress and should provide concrete proof of concept or initial analysis with respect to your research questions.

Your results should be specific about exactly what data were used and how the results were generated. For example, if you filtered out some of the data due to A and B reasons, you should state what criteria were used to filter the data, why, and how much of the data was filtered out (or is left). These steps should be explained in enough detail such that an informed reader (like another group working on the same data set) could reasonably be expected to reproduce your results with time and effort. Just saying, for example, “we cleaned the data and dealt with missing values” is not sufficient detail.

Checklist for section

  1. The section clearly shows substantial effort has been made since the proposal
  2. Clear that the data has been loaded and at least preliminarily processed
  3. Sufficient detail on how a result was generated

Part 4: Reflection and Next Steps

In this part, you should answer the following sections in their own subsection:

  1. Successes/Mostly Complete – What has been successful in the project so far or what is essentially complete and ready for the final report? How to access the data, code, or other supplementary resources that you have.
  2. Challenges/Incomplete – What has been challenging in the project so far or what is incomplete in the prototype that needs to be finished for the final report?
  3. Collaboration plan reflection – How is the collaboration going? What is currently happening versus the original proposed plan? Is the group okay with what is happening? Does the group need to renegotiate what the plan should be? If yes, what is the new plan?
  4. Next Steps – What are your next steps? These should be concrete and specific actions that your group will take to address the challenges identified in order to complete a successful final project.

Checklist for section

  1. Subsection: Successes/Mostly complete (and states how to and Prof. Stephens-Martinez can access everything)
  2. Subsection: Challenges/Incomplete
  3. Subsection: Collaboration plan reflection
  4. Subsection: Next Steps

Feedback and Grading Rubric

Each section will be graded on a four-step rubric scale as follows.

  • E (Exemplary) – Work that meets all requirements of that section.
  • S (Satisfactory) – Work that meets all requirements with only slight mistakes or missing pieces of information.
  • N (Not yet) – Work that does not meet some requirements and/or displays developing or incomplete work that needs substantial revision to meet satisfactory standards.
  • U (Unassessable) – Work that is missing, does not demonstrate meaningful effort, or does not provide enough evidence to determine a level of mastery.

The entire assignment is worth 100 points.

  • 12 points will be allocated for meeting general directions (length, on-time pdf submission, group submission, etc.). You cannot submit a prototype greater than 4 pages (not counting references). Learning how to be succinct is an important skill.
  • 22 points are allocated for each section. (22*4 = 88)

The rubric will be converted to points as follows:

  • E = full credit
  • S = E_full_credit – 1
  • N = E_full_credit / 2
  • U = E_full_credit / 5
  • Blank = 0

Anything earning less than an E will receive feedback in Gradescope (and E’s may also get feedback). If your submission earns less than an S in any section, you will be allowed 2 resubmissions to bring it up to the E or S standards for all sections. If your proposal earns E’s and S’s only, you can have 1 resubmission if your group decides to aim for a higher score. Each resubmission must be done within 1 week, starting from when the feedback is returned. This is to limit the amount of time spent on the assignment for all those involved.

Project Proposal

Due: Thursday 9/29

General Directions

The purpose of this document is to prepare your team for success in the course project. Your proposal should contain at least three parts, which we define below. In terms of length, it should be 2-3 pages using standard margins (1 in.), font (11-12 pt), and line spacing (1-1.5). In addition to these three components, you should provide any additional context or information necessary to understand your vision for your project. You should convert your final document to a pdf and upload it to Gradescope under the assignment “Project Proposal” by the due date. Be sure to use the group submission feature on Gradescope to include all of your group members in a single submission.

Introduction and Research Question

Your proposal should introduce your topic in general and motivate why your research question is relevant. Relevance addresses the importance and interest of your research question to the CSEd community (or at least this class). It should then define one or more research questions. The research question(s) should be substantial and feasible.

  1. Substantial research questions require more than a surface-level analysis (more than just computing basic summary statistics on readily available datasets, for example).
  2. Feasible research questions can actually be addressed by your group over the course of approximately eight weeks, including writing a report and creating a presentation.

Related Work with Status

Your research questions should be informed by related work. This section should summarize the key takeaways from the related works you have read so far with a note on the reading pass level for that work (see the reading academia papers assignment). There should be at least 3 works found by the time of the proposal that have been read at least at a pass 1 level. And at least 4 more that will potentially be useful but only the title and abstract have been read to determine their potential relevance.

The works do not need to be very similar to the research question you are proposing. It is okay if they simply motivate why your research question is relevant (For example, if your research question is about UTAs, a related work could be how novices struggle to learn computer science material, which motivates why UTAs are important).

Overall, for a related work section in any project report, not all of your related work needs to be read at the pass 3 level. In fact, most of them will not need that level of a pass. Only a handful will need to be read at a pass 3, a few more at pass 2, and most of them will be at a pass 1 level. How many fall in each bucket will depend on how similar a given related work is to your research question.

Collaboration Plan

While working in pairs or triples usually does not require a lot of coordination, having a frank conversation on working styles, communication expectations, etc. is important for any team. Therefore create a plan that addresses the following:

  1. How will you divide responsibilities? Will some students be responsible for certain portions of the project, or will you be more integrated and decide on responsibilities every week?
  2. About how much time do you expect every group member to spend on the project each week, on average? It is ok if this number is higher toward the last couple of weeks of the semester.
  3. When and how will you meet? You should plan to meet at least once per week for at least 30 minutes to check in on one another’s progress, get help, and plan for what comes next. Identify a day of the week, a time, and the platform you will use to meet.
  4. What platform(s) will you use to communicate between meetings? Will you primarily use email, text, slack, or other chat apps? If you want a more professional enterprise tool, Duke provides free access to Microsoft Teams.
  5. Where will you store data, code, writing, etc., so that all group members can easily access shared materials? Duke provides free access to Box and GitLab, which could serve these purposes, but you could also use external services like Google Drive or GitHub. Provide a link to the folder/repository in your proposal to demonstrate that it is created and ready. Remember that data should only be on Duke-provided tools (e.g. Box) and never in a version-controlled repository.

Feedback and Grading Rubric

Each section will be graded on a four-step rubric scale as follows.

  • E (Exemplary) – Work that meets all requirements of that section.
  • S (Satisfactory) – Work that meets all requirements with only slight mistakes or missing pieces of information.
  • N (Not yet) – Work that does not meet some requirements and/or displays developing or incomplete work that needs substantial revision to meet satisfactory standards.
  • U (Unassessable) – Work that is missing, does not demonstrate meaningful effort, or does not provide enough evidence to determine a level of mastery.

The entire assignment is worth 100 points.

  • 10 points will be allocated for meeting general directions (length, on-time pdf submission, group submission, etc.). You cannot submit a proposal greater than 3 pages. Learning how to be succinct is an important skill.
  • 30 points are allocated for each section.

The rubric will be converted to points as follows:

  • E = full credit
  • S = E_full_credit – 1
  • N = E_full_credit / 2
  • U = E_full_credit / 5
  • Blank = 0

Anything earning less than an E will receive feedback in Gradescope. If your proposal earns less than an S in any section you will be allowed 2 resubmissions to bring it up to the E or S standards for all sections. If your proposal earns E’s and S’s only, you can have 1 resubmission if your group decides to aim for a higher score. Each resubmission must be done within 1 week, starting from when the feedback is returned. This is to limit the amount of time spent on the proposal for all those involved.