The Brewing Standards War - Verification Methodology

Back in August when the OVM was announced one of the big unknowns was how Synopsys was going to respond.  If Karen Bartleson's recent post on standardizing verification methodologies is any indication, the response is going to be to attempt to slow down and gain some element of control of the OVM via a drawn out battle in an Accellera standards committee. 

To be perfectly frank, once VCS supports enough of the SystemVerilog standard to run the OVM (and I can't believe they'd be stupid enough not to be working on such capabilities) this whole discussion is a moot point.  Synopsys had a chance a few years back to release the VMM and maintain some sort of technical leadership in the area of verification methodologies.  Sadly, they chose instead to keep their library to themselves, sowing seeds of doubt in VCS users who might have adopted the VMM if not for the worry that it would be difficult, if not impossible to run it on competing simulators.  In addition, they've let things slide a bit recently to the point where the VMM library simply doesn't have the same robust capabilities as the OVM. 

For example, the base VMM library doesn't scale well from module level to full chip test environments, due to the restrictive nature of the vmm_env top level object.  They also tend to require (again, due to the way envs are constructed) a very flat hierarchy of objects in the testbench, making it inconvenient to encapsulate things in a way that makes it possible to reuse large collections of objects.

Another area where the VMM is lacking is in its inability to easily support the "virtual sequence" concept supported by the OVM.  On a recent VMM-based project I had to implement my own virtual sequence infrastructure to deal with test scenarios that needed to procedurally access several drivers within the environment.  Why should I have to spend time coding something this basic?  The concept has been around since the early days of the eRM back in 2002.

Basically, over time, Synopsys has grown complacent with their perceived leadership in the area of SystemVerilog methodology libraries, and now finds themselves in the unusual position of having to play catchup to the Mentor/Cadence coalition.  I understand that business is business, but if Synopsys is serious about being involved in a verification methodology standards effort they should start by releasing a version of VCS that supports the constructs required to run the OVM and release the source of the VMM so that other simulation vendors can incorporate any necessary changes into their platforms.  That way the industry can openly evaluate the merits of both technologies and have an honest, user driven technical discussion about the pros and cons of various approaches to testbench design without the typical marketing BS. 

Comments

[Janick Bergeron]

I would like to make some factual corrections to your article.

1) We *did* release VMM a few years back: we published the specification in a book that anyone could buy without any license or NDA. What we have not generally released is our implementation of the VMM Standard Library (however it is important to note that we do make it available to VCS customers and EDA vendors in the VMM Catalyst Program). The other EDA vendors chose to re-invent VMM (I'll spare you the litany of acronyms that went into the re-invention process) instead of implementing VMM and focusing on value-add functionality (see #2). Open-sourcing does not automatically create a standard (remember TestBuilder?), nor does a standard require open sourcing (see your SystemVerilog simulator).

2) We have not let VMM slide nor grown complacent. At DAC last year, we announced a series of VMM applications that provide even more functionality and productivity and have delivered on all of those announcements. See http://www.edn.com/article/CA6531592.html . And we have a clear roadmap for additional VMM capabilities.

3) There is a new VMM Environment Composition application (vmm_subenv) that addresses the module-to-SoC scalability issue. It addresses both the "restrictive nature" of vmm_env (which was designed for 1st level reuse i.e. across different testcases, not 2nd level reuse, i.e. across different testbenches) and the self-imposed flatness of the hierarchy by providing a container for collection of testbench elements (I say "self-imposed" because there is nothing in the VMM methodology that prevents a user from creating their own intermediate hierarchy).

4) As for the "virtual sequence" concept, we haven't seen explicit demand for something like it from the hundreds of VMM users over the last 3 years. The VMM scenario generator appears to meet their requirements. It may seem like this concept is missing from VMM if you come from an eRM background because there isn’t a one-to-one correspondence, but the same goals can be achieved in VMM using the existing infrastructure. However, should you wish to contribute your "VMM Virtual Sequence" application to the VMM user community, we'd be happy to make it available to anyone wishing to use it.

Cheers!

[Ambar Sarkar]

JL,

From a user perspective, I have to say I like the idea of a single standard. However, here are my specific thoughts..

1. We need BOTH a standard and an open reference implementation. Just a paper spec does not let me use the standard. Imagine someone handing me the manual for the C++ template library(STL) but not the library. OTOH, having the standard enables innovation on top of it, including other, potentially more efficient implementations. Most importantly, an open reference implementation empowers the verification community as a whole, as we have seen with Linux.

2. I believe the MOST critical issue today is the implementation of the SystemVerilog standard fully. Just that alone would eliminate many of the issues that vex the user community today.

3. I believe the standardization body should consider both VMM and OVM as potential candidates. As an end user, I believe one can be productive using either approach, and these methodologies are still evolving.

4. The community may not have the stamina or patience for morphing VMM/OVM into yet another methodology. If need be, we may have to live with two standards, as in SVA/PSL. I hate that idea, but in that case, the marketplace will determine the rest.

Cheers,
-Ambar

[Dennis Brophy]

Janick,

Perhaps Synopsys can post their customer & partner license terms and conditions. Since Synopsys prefers to keep this information hidden, it is difficult for the market to judge Synopsys' sincerity to foster collaboration. For a company that is so open and has nothing to hide, it hides everything for VMM.

If the licenses were open, others could conclude what I have concluded: Synopsys does not make VMM available under consumer friendly terms and conditions.

-Dennis

[Scott Roland]

I think that Ambar makes some good points about the limitations of a specification alone and the need for a full implementation of SystemVerilog.

The VMM book is about 2.5 years old and the IEEE-1647 e standard is about 2 years old, but so far neither has resulted in an alternative implementation. The biggest challenge a vendor would face if they were to support either one would be ensuring that it works with customers existing codebases. One of the benefits of the OVM is that it is a publicly accessible library of code that vendors can test their implementations against. Still, even with the OVM one gets a sense that Cadence and Mentor still have some issues with their SystemVerilog support. But one can hope/assume that that in the near term the OVM and its coding style will work with those vendors (and hopefully more).

[Adrian Coman]

TrustIC has released a few years ago a simulator independent VMM library which is working with all major simulators: NCSim, Questa and VCS. You can read more about it here: http://www.trustic.eu/vmm-library.html

In order to ease even more the adoption of VMM in any verification environment, regardless of the simulator, we are now in the process of releasing our VMM library under the GPL license, making it freely available for all SystemVerilog users, just like OVM. We hope that having both methodologies freely available will help the verification community make a better choice on what is more suitable for their verification needs.

[Paul Marriott]

Adrian,

I'd caution against releasing your VMM library under the GPL - there were many good technical reasons why the OVM source was released under an Apache 2.0 licence, not the GPL.

Previous discussions about VMM being open (or otherwise) versus the source code of the simulator are somewhat moot - SystemVerilog is an open standard - a user doesn't care how a particular implementation works, it's whether it adheres to the published standard that's important.

The problem with methodology class libraries is that they are effectively used to encapsulate a verification engineer's verification IP - it is for this reason that open source is important so that the IP developer is free to control all of their IP. As much as there might be published a guidebook for, say, the VMM, there is no implementation specification with which one can be confident that one's IP will behave correctly on any simulator other than the methodology class library's vendor's. This is the crux of the matter and why a donation to Accellera is probably irrelevant now that the OVM code is open source: effectively the implementation specification is in the source itself.

All this said, it could be interesting if the "plumbing" aspect of the VMM could be based on the OSCI TLM standards, like the AVM and the OVM. This would pretty much guarantee interoperability and would leave users much greater freedom to chose which methodology layer to use on top of the basic infrastructure - this and aligning the reporting infrastructures. Of course, if the VMM was open source, it would be easy for users to do this should they feel like it ;)

[JL Gray]

FYI, my response ran a bit long, so I decided to publish it as a separate post:

http://www.coolverification.com/2008/02/vmm---robust-an.html

As always, comments welcome!

JL

[Karen Bartleson]

Dennis,

I've addressed your concerns on my blog at: http://www.synopsysoc.org/thestandardsgame/

Synopsys is not as sinister as you imply. :)

Karen