VAST Scientific provides independent, analytically-rigorous evaluation of an emerging technology system
throughout its development stages. At early stages, strategic assessment ensures that the system
under design is focused on meeting identified industry capability needs and is adoptable by users into
their operating environments. At later stages, measurement-based assessment reports provide
developers, financiers, and other stakeholders with defensible knowledge on the current status of the
venture and of the issues requiring remediation to meet venture goals.
Modern information-driven systems are complex, involving integrated commercial-off-the-shelf (COTS) and
private source software components supporting data management and information extraction across a
distributed environment of privilege-managed shared services and products that must meet the needs of a
diverse user population. While components may be rigorously tested prior to integration, it is
essential to thoroughly test their performance in the context of system operations. This provides
incredible insight into performance bottlenecks and their impact on the effectiveness of the system to
its users. The system must also be assessed under realistic deployment strategies and user
loads. While common practice, it is inadequate to use limited functional demonstrations as a
measure of venture status.
We propose that an integrated and comprehensive assessment strategy can streamline the development
process by ensuring capability benchmarks are met for a realistic operational environment.
Strategic assessment services include:
- Research system capabilities are matched to documented industry needs, and propose technology
candidates that can support these capabilities (initial architecture support).
- Decompose the proposed system architecture into its key functional components to identify a
comprehensive suite of metrics (measures of performance and measures of effectiveness) needed to
perform a thorough assessment.
- Design a data collection and analysis plan that supports the metric assessment, including the
operating environment and user interactions with the system.
- Identify key system data collection points for each functional component and coordinate with
developers to insure appropriate data capture instrumentation. As a leave-behind, this can
provide a long term benefit to the administrating agency, who can continue to mine these system data
feeds for component or node performance, user usage, and resource management beyond the assessment
- Execute assessment data collection events in a real-world or representational environment, including
the capture of subject matter expert opinion on apparent performance (lessons learned).
- Analyze system data against measures of performance and assess system effectiveness by comparing
performance to user and community expectations.
- Report assessment findings coupled with recommendations for remediating identified issues.
Assessment findings are then used to drive the next development cycle.
To promote effectiveness, it is essential that VAST Scientific assessors work closely with system
developers. Strategic assessment is not antagonistic in nature, as it is designed to document the
successes as well as the issues requiring remediation, and includes the expert opinion of the
developers. However, objectivity is maintained through its basis in scientifically defensible
measurements obtained through our third party testing and analysis.
If objectivity is not required, we can design an evaluation plan that our customer can then execute
[ back to top ]