Subject Matter Expert Reviewers Needed – Integrated Pest Management (IPM) Question Evaluation

Click Here to Apply

Overview

The Extension Foundation is seeking Integrated Pest Management (IPM) subject matter experts from the Cooperative Extension System, USDA, and related agricultural research organizations to assist with the evaluation of a benchmark question set used to assess an AI-based agricultural advisory system.

This work supports the development and evaluation of ExtensionBot, an AI-powered conversational assistant designed to provide research-based pest management guidance grounded in Extension publications and USDA resources.

The goal of this project is to ensure that the system accurately reflects real-world agricultural questions and decision contexts faced by farmers, crop advisors, and Extension educators.

Your expertise will help validate whether the questions used to test the system are realistic, relevant, and representative of common IPM challenges.

Project Background

ExtensionBot is a conversational AI system designed to help farmers and agricultural professionals access research-based pest management guidance drawn from Extension and USDA publications.

Unlike general-purpose AI tools, ExtensionBot uses a retrieval-based architecture, meaning its responses are grounded in trusted Extension and research content.

To ensure the system performs well in real-world agricultural scenarios, we are building a benchmark evaluation dataset consisting of approximately 100 IPM-related questions.

These questions are used to test how well different AI systems answer common pest management inquiries.

Before using this dataset for formal evaluation, we are asking IPM experts to review the questions themselves to confirm that they represent authentic, reasonable, and meaningful agricultural decision scenarios.

Role of the Reviewer

Subject matter experts will review a set of approximately 100 IPM-related questions and provide feedback on their quality and appropriateness.

Reviewers will not be evaluating AI-generated answers.
Instead, reviewers will assess whether the questions themselves represent realistic agricultural inquiries.

For each question, reviewers will be asked to consider:

• Is this a reasonable question that a farmer, crop consultant, or Extension client might ask?
• Does the question represent a valid IPM scenario or diagnostic situation?
• Is the wording clear and understandable for an agricultural audience?
• Does the question reflect a meaningful pest management decision or concern?
• Are there any questions that appear misleading, unrealistic, or poorly framed?

Reviewers may also flag questions that:

• need clarification
• should be revised
• should be removed from the evaluation dataset

Why This Work Matters

Artificial intelligence systems are increasingly being used to provide agricultural information and decision support.

However, evaluating these systems requires carefully designed test questions that reflect real-world agricultural practice.

By validating this question set, IPM experts help ensure that the evaluation process measures what truly matters:

• realistic pest identification challenges
• appropriate treatment decision contexts
• integrated pest management principles
• regulatory and pesticide safety considerations

This expert review helps ensure that the evaluation framework reflects the realities of agricultural production and Extension practice.

Scope of Work

Reviewers will:

• Review approximately 100 IPM-related questions
• Provide simple ratings or feedback on each question
• Flag questions that are unrealistic, confusing, or inappropriate
• Suggest edits or improvements when helpful
• 2-3 Zoom meetings

Estimated time commitment:

8-12 hours

Deliverables

Reviewers will submit:

• A completed evaluation form based on a formal rubric
• Optional written comments or suggested revisions for specific questions

Desired Expertise

We are seeking reviewers with experience in areas such as:

• Integrated Pest Management
• Crop protection
• Plant pathology
• Entomology
• Weed science
• Agronomy
• Extension crop advisory services
• Experience working directly with farmers, crop consultants, or agricultural producers is particularly valuable.

Compensation

Participants will be compensated with a $1000 honorarium.

About the Extension Foundation

The Extension Foundation is a nonprofit organization that supports the U.S. Cooperative Extension System by developing technology, tools, and programs that help Extension professionals better serve agricultural communities.

This project builds on Extension Foundation’s work to develop ExtensionBot, a conversational AI system designed to provide accessible, research-based agricultural guidance grounded in Extension expertise.

Click Here to Apply