VMM - Robust and Open, Stale and Proprietary or Somewhere In Between?

Last week I wrote an article about what appears to be an upcoming standards battle pitting the OVM vs. the VMM.  My comments seem to have touched a nerve with a large number of people and prompted comments from Janick Bergeron of Synopsys, Dennis Brophy of Mentor and Ambar Sarkar of Paradigm Works, among others.  Since this thread is starting to take the form of an all out smack-down, I figured perhaps the readers of this site could help me pick out appropriate avatars for Janick, Ambar, Dennis, and myself.  Figured out who's who?   Good, now let's continue!

Janick was concerned that my argument contained some factual errors - specifically with my assertions that:

1. The VMM is not open.
2. The VMM is not as feature-rich as the OVM. 

Ambar commented on the need for an open reference implementation of any potential standard verification methodology library.  He also bemoaned the lack of support for the SystemVerilog standard among the EDA vendors, and ended with a comment that the verification community "may not have the stamina or patience for morphing VMM/OVM into yet another methodology." 

Finally, Dennis challenged Synopsys to "post their customer & partner license terms and conditions," the implication being that Synopsys uses restrictive licensing agreements with its customers. 

I'd like to take some time to address Janick's concerns.  Though I haven't used it myself, I believe Janick is correct that the new "VMM Environment Composition" component deals with some of the issues preventing reuse of module level verification components.  It is also true that Synopsys has announced (and has released to some customers in beta form) several additional components of the VMM. The reason I didn't mention them in my original post is that for the most part, they are not currently included in shipping versions of VCS, nor are they, to my knowledge, fully documented publicly. 

I stand by my original statement though about the the fact that Synopsys has "let things slide a bit recently to the point where the VMM library simply doesn't have the same robust capabilities as the OVM."  Why?  With the exception of the register abstraction layer, the new VMM capabilities Janick described are already part of the OVM.  For example, the vmm_consensus object roughly corresponds to the  objection mechanism in the OVM (look up documentation on ovm_component::global_stop_request() for more info). The flexibility provided by the vmm_subenv appears to be roughly similar to the use of an ovm_component to wrap up the reusable portions of a module level environment, but even then, it cannot match the flexibility provided by the ovm_factory object.  Basically, in an environment that takes advantage of ovm_factory, a user can swap in and out different versions of objects in the environment (derived children) at runtime without requiring any changes to the underlying component.  That's a powerful approach that can only be duplicated in the VMM with user-defined additions to the the library.

Speaking of user-defined additions, I'd also like to make a comment on Janick's assertion that virtual sequences have not been requested by the VMM user base.  Virtual sequences are very different from the VMM scenario generator.  The scenario generator (to the best of my knowledge) is used to generate transactions which are then sent to a corresponding driver.  However, what is a user meant to do if the transactions need to be sent to multiple drivers and potentially contain procedural logic that cannot easily be encapsulated in a vmm_data object?  I've spoken with folks from Synopsys about this, and the way I understand it, people tend to implement their own custom solutions on top of any scenario generators they may have created for their environment.  That sounds suspiciously like the concept of virtual sequences to me.  If I've misunderstood the purpose of the scenario generators though please let me know!

Finally, let's talk about the openness of the VMM.  Let me start by saying that I'm not a lawyer and am happy to be corrected by someone who has more experience in this area than I do.  As best as I can tell, Synopsys and ARM have retained full copyright to the VMM and, I'm assuming, to the API described therein.  To me, that means anyone can write code that uses the API, but that recreating the underlying VMM implementation itself would require a license from ARM and Synopsys.  That would mean that, contrary to Janick's assertion and separate from any discussion of whether the VMM should be open-sourced or not, the VMM is closed to development without getting a license agreement and potentially paying royalties.  This brings us to the point that Dennis made about the availability of license information.  I honestly have no idea what it would take for Synopsys to allow me to create my own implementation of the VMM.  And, at this point, I see no value in doing so given that I could download a copy of the OVM and create derivative products without requiring a licensing deal. 

As I was writing this, Adrian Coman from TrustIC (authors of a Synopsys-independent version of the VMM) and Paul Marriott of XtremeEDA chimed into the discussion.  Paul makes a very good point that the VMM is not specified well enough to ensure that competing implementations all would behave in the same fashion.  His other point?  That the OVM is effectively a reference implementation that is complete enough to define a standard. 

My question for Adrian is, do you have a license to create/sell your implementation of the VMM?  If not, are you concerned that you might run into difficulties?  As I said previously, I don't fully understand how other VMM implementations could be created without dealing with licensing issues.

What do the rest of you think?  Is your company likely to use the OVM?  To stick with or start using the VMM?  Are you one of those companies (I know of a couple at least) who refuse to use vendor verification libraries unless they are open source?  Are you interested in a drawn out standards battle between the VMM and OVM or are you likely to ignore the whole thing and just pick one? 

Comments

[Karen Bartleson]

JL,

IANAL either, but once Synopsys is able to donate our implementation to Accellera, the whole licensing argument becomes irrelevant.

Then, we can start arguing which technical aspects of VMM and OVM are better or worse in a formal setting as we work towards a single standard.

Karen

[David Maliniak]

Karen:

Yes, but did you, or did you not, steal Dennis Brophy's hair?

In all seriousness, I'd like to see all of the vendors converge on a *truly open* methodology. Then users like JL can judge tools on an apples-to-apples basis on technical merit, correct?

[EDA tool user]

In my life, everything closed source has caused tremendous problems (bad performance, incorrect operation, etc.), that despite promesses of support, and despite my employers paying $$$ in support. In closed source, all bugs are kept secret, users cannot and do not learn from one another.

Have you ever hear "oh my, you are the first customer to report this problem! - But rest assured, we already have an undocumented environment variable that you can specify to work around the problem - i.e. you are not really the first customer to complain but we like to lie to you because we can: your entire flow is within our hands (ie your balls are in our vice)."

The close source binary encrypted proprietary software business is a wonderful way to make tons of money and control customers. They will *never* give up making money - you want your synopsys shares to be worth a lot don't you? However, they could open the bug databases to their users. Right now they call it "solvnet", but it's only the bugs you reported that you see, not all the bugs reported by all the customers. There is nothing like seeing the real bug reports from the real developers to know what the software can and cannot do. And I bet they would save money doing that too: think of all the first line support that would vanish if we could search the bug databases the internal developers really use... they do use a bug database internally right?

To go back to my first sentence, I am not saying open source is better in terms of performance and correctness (well, sometimes it is), but I am saying that in open source there is no secrecy about bugs and malfunctions, and users are able to develop faster when they know the bugs. In closed source bugs are kept secret and are surrounded with red tape.

[JL Gray]

EDA tool user,

Gee, you're not bitter or anything, are you? ;-). You make an interesting comment about the openness (or lack thereof) of bugs for EDA tools. Before I'd care at all about seeing an open source version of VCS, IUS or Questa I'd care about seeing *all* of the bugs. As you've pointed out knowing that info can save significant amounts of debug time. There are "closed source" companies that release bug reports though, so I tend to attribute the lack of openness about these types of things to the general culture of EDA companies (i.e. keep things secret, marketing controls what gets published to the outside world, etc).

JL

[HarryG]

Karen makes a great point. I spent 14.5 years in the EDA industry and have witnessed several standards come to be in that time. Some are vendor generated and then donated (e.g. SDF donated by Cadence) and become standards. Some evolve more organically (e.g. IEEE 1149.1) through the standards committees. Ultimately, there needs to be some form of industry committee, consisting of many stakeholders, for a standard to be truly open. Here is why...

I asked one of the marketing managers from Mentor the other day whether there were any plans to donate OVM to a standards committee like Accellera. They don't. "So", I asked, "who gets to decide how the standard evolves"? "We do", that is, Cadence and Mentor. "What about the customers actually doing verification, do they have a say?" Silence.

Not 5 minutes later, he was telling me how very little came into OVM from Cadence and that it's mostly AVM, trying to lay the FUD that Cadence does not have much to offer and that Mentor really pulled most of the weight. This is the guy who I'll be trusting to decide what goes into the next generation of OVM?

It's not just EDA. Many standards committees are composed of member companies lobbying for the standard to include some detail that gives them an advantage. That's the game...getting the standard close to what you can do well so you can come out with a more competitive product.

As long as an EDA company (or two) control the standard, its going to be used more to their benefit and competitive advantage than to the benefit of verification engineers. I say, if you love OVM so much, then let it go.

[harry the ASIC guy]

Looks like the next arrow in the Standards war has been fired by Synopsys (see http://synopsysoc.org/thestandardsgame/?p=64 ). What's the next move going to be from Mentor and Cadence?

[gruntled_verifico]

I wrote a multi-simulator systemVerilog environment in early 2006
(Cadence and Synopsys). During the course, I found that

1. One common requirement, parameterized classes, were not supported
either by sim-A or sim-B. Lots of tickinc'ing later, I hacked up a
horrible workaround.

2. Another common requirement, encap global types and methods in
global classes can be done in sim-A and not sim-B. So no global
classes. But if I tried to package it, than if worked in sim-B and not
sim-A. Lots of ifdef SIM-A, ifdef SIM-B, I got it rolling again.

3. I tried throwing strings into the game and host of features in both
sim-A and sim-B caved. (fgets works, but not sscanf, except this and
that and here and there).

There were many other issues which I wont go into, except to say this
was not a very complex DUV, but still unearthed these discrepencies.

Last week (2008), I started another project in another company, and I
see pretty much the same issues.

SV support is at this state and they already built methodology
libraries on top of it? And they don't interoperate? And we are
surprised? It feels like a methodology sky scraper is being built on
top of colliding system Verilog tectonic plates. And I am King Kong on
top, desperately trying to protect my fragile DUV, while bugs rain
down on me.

As SV support matures across all simulators, will the methodologies be
revisited? Or will it be a microsoft approach (more features better
than fewer bugs).