Why Most Agency Vetting Fails
The typical procurement process for a development agency involves looking at a portfolio, reading a few testimonials, and comparing price quotes. This tells you almost nothing about whether the agency will be a good partner for your specific project. Portfolios show you what an agency has shipped — not how they handled scope changes, technical disagreements, or delivery delays. Testimonials are curated. And price comparisons without scope alignment are meaningless.
The questions below are designed to surface the things that actually predict a successful engagement: how the agency manages uncertainty, who will actually build your product, how they handle the end of a project, and whether they will be honest with you when something is not going well.
Category 1: Process & Communication (Q1–Q3)
How an agency runs a project operationally tells you a great deal about whether your experience will be smooth or chaotic. These three questions reveal the structure, or lack of it, behind the scenes.
Why it matters: A good agency has a repeatable, documented process. They should be able to describe it clearly — discovery phase, design review, development sprints, QA cycles, staging review, launch protocol. Vagueness here is a warning sign. If they cannot articulate their process before the work starts, they are improvising it as they go — and that improvisation becomes your problem when things get complicated.
They describe a structured process with named phases, explain who is responsible for what, and can tell you where client input is required and when decisions get made.
They talk in generalities ("we're collaborative and flexible") without describing any concrete process steps, milestones, or decision points.
Why it matters: Scope changes are inevitable on almost every real project. The question is not whether they happen — it is whether the agency has a clear, fair process for handling them. Agencies without a change management process either absorb everything silently (building resentment) or surprise you with large invoices later. Neither is good. A mature agency will have a written change request process with clear timelines and pricing implications.
They describe a formal change request process: new requirements go through impact assessment, you get a written estimate of time and cost, and both parties agree before work proceeds.
They say "we'll figure it out" or "we're flexible, don't worry about it" — which usually means they will absorb small changes silently until they hit a breaking point, or bill you for changes you didn't expect to pay for.
Why it matters: Communication failures are the most common reason client-agency relationships break down. Not technical problems — communication failures. You need to know: how do you raise an urgent issue at 4pm on a Thursday? Who responds to it? What is the expected response time? How are decisions documented? Agencies that run projects well have answers to these questions ready before you ask them.
They name a specific point of contact for you, describe a regular cadence (e.g., weekly status call, shared project management tool, async updates on Slack or similar), and explain how they document decisions.
They say "we'll be in touch whenever you need us" with no defined structure — or they promise responsiveness without describing any system that makes it reliable.
Category 2: Technical Capability (Q4–Q6)
Portfolio entries look impressive. What you need to understand is whether the technical depth behind them is real and whether it matches what your project actually requires.
Why it matters: In many agencies — especially larger ones — the people who sell the work are not the people who build it. You may be impressed by a senior partner in the sales meeting, then handed off to a junior developer you have never met. Asking to meet the lead developer before signing is a completely reasonable request, and how the agency responds tells you a great deal. A confident agency welcomes this. An agency that resists is often protecting something.
They arrange a brief introduction with the developer or team lead who will be assigned to your project, and that person can speak specifically about your requirements.
They deflect ("our developers are very busy but you'll meet them during kickoff"), which often means the staffing decision has not been made or the people available are not as experienced as the sales team implied.
Why it matters: A polished screenshot in a portfolio proves very little. A live product that you can interact with proves substantially more. Ask for live URLs. Ask whether you can see the application running, not just designed. And specifically ask whether they have experience with the technical components your project requires — if you need real-time features, complex integrations, or a specific compliance requirement, you want to know before you sign whether they have actually done that before.
They can share live URLs of comparable projects, speak specifically about the technical challenges involved, and describe what they would do differently on your project based on that experience.
They can only show screenshots, or the live examples they share do not have the technical characteristics your project requires — suggesting the similarity is cosmetic rather than functional.
Why it matters: Many agencies treat QA as an afterthought — something they do in the final week before launch, under time pressure, with whoever is available. This produces software that passes a basic click-through test but fails under real usage conditions. A serious agency builds QA into the development cycle, not just the end of it. They should be able to tell you what types of testing they perform (functional, regression, cross-browser, performance) and who is responsible for each.
They describe a structured QA process with named responsibilities, describe the types of testing performed at different stages, and ideally have a dedicated QA resource rather than relying on developers to test their own code.
They say "we test everything before we deliver" without any specifics, or testing appears to be primarily manual click-through with no systematic approach to regression or edge cases.
Category 3: Business Fit (Q7–Q9)
Technical capability is necessary but not sufficient. The agency also needs to be the right size, right structure, and right commercial fit for your engagement.
Why it matters: An agency that is managing fifteen projects simultaneously with a team of ten people is not going to give your project sustained attention. Ask directly. You want to understand capacity. A project that is important to you may be one of many to them, and delivery quality degrades when teams are stretched thin. Some agencies are honest about this and plan accordingly — others overpromise and underdeliver.
They can give you a specific number, explain how they manage capacity, and describe how they ensure your project gets dedicated attention rather than competing with others for developer time.
They are vague ("we manage our capacity well") or the math doesn't work — ten team members, twenty active projects, and a four-week timeline for your engagement is a warning sign regardless of what they tell you.
Why it matters: Software doesn't stop needing attention at the moment of launch. There will be bugs found in production, features you want to add, dependencies that need updating, and infrastructure that needs monitoring. If the agency has no post-launch support structure, you are either locked into them at whatever rate they decide to charge, or you are starting a search for a new developer with no one maintaining your codebase in the meantime. Knowing the post-launch relationship structure before you sign is essential.
They have a defined post-launch support offering — whether a retainer model, a support ticket system, or a formal handover process if you want to take it in-house — with clear pricing and SLAs.
They have not thought about it ("we'll cross that bridge when we come to it") or they are only interested in new projects and have no support infrastructure for launched products.
Why it matters: Every agency can give you references from satisfied clients. Those references tell you about the best-case version of working with this agency. What you want to know is how they behave when things go wrong — because on a sufficiently complex project, something almost always does. Asking specifically for a difficult reference is a revealing request. Agencies who can provide one and discuss it candidly are demonstrating the kind of honesty that is valuable in a partner. Agencies who cannot are either hiding their failures or have never reflected on them.
They can name a specific difficult engagement, explain what happened honestly, describe how they responded, and provide contact details for the client.
They claim a perfect track record ("all our clients are very happy") or produce only pre-screened references who were never involved in a challenging project.
Category 4: Risk & Transparency (Q10–Q12)
These three questions get at the issues that tend to surface only when a project goes wrong — but by then it is too late to address them proactively.
Why it matters: IP ownership should never be assumed — it should be documented clearly in the contract before work begins. Some agencies retain ownership of the code until final payment is made (which is legitimate but should be disclosed). Some retain rights to certain components (which may or may not be acceptable depending on your situation). The important thing is that there is no ambiguity — you should be able to read the contract and know exactly what you own and when.
The contract clearly assigns all IP to you upon delivery and payment, and the agency is transparent about any exceptions (e.g., licensed third-party components or proprietary frameworks they retain rights to).
The IP language in the contract is vague, or the agency deflects the question ("don't worry, the code is yours") without pointing to specific contract language that confirms it.
Why it matters: Timeline overruns happen. The question is whether the agency has a transparent, structured way to handle them — or whether they either hide the overrun until it is unavoidable, or invoice you for the additional time without prior discussion. A mature agency will proactively raise timeline concerns as soon as they become apparent, present options for how to handle them (descope, extend timeline, add resource), and get written agreement before proceeding.
They describe a proactive escalation process: when they identify a risk to the timeline, they flag it immediately, present options with cost implications, and require written approval before any changes to scope or budget.
They promise the timeline is fixed with no acknowledgment that overruns are a real possibility — or they are vague about what happens, suggesting they deal with it case by case rather than having a structured process.
Why it matters: The handover at the end of an engagement is where many agency relationships break down — and where you discover whether the agency actually cares about your long-term success or just about delivering the invoice-triggering milestone. Good handovers include a README, architectural documentation, deployment runbooks, credentials handover, environment documentation, and often a walkthrough call with the incoming developer or your team. Bad handovers are a GitHub link and radio silence.
They have a documented handover checklist, describe the specific artifacts that will be produced, and offer a handover session where they walk through the codebase and infrastructure with whoever is taking over.
They treat handover as outside the scope of the engagement ("we deliver the code, what you do with it is up to you") or cannot describe what documentation they typically produce.
If you only enforce three of these twelve questions, make them these:
- Meet the actual developer who will build your product before signing.
- Get IP ownership language confirmed in writing in the contract — not verbally.
- Understand the change management process before the project starts, not when the first change request arrives.
One Final Thing Worth Saying
A good agency will not be offended by any of these questions. In fact, the best ones will have clear, immediate answers — because they have been asked before and they have built systems to handle them. If asking any of these questions creates friction, discomfort, or vague deflection, that is information too. The negotiation before the contract is the easiest conversation you will ever have with an agency. Pay attention to how it goes.