Ascend Archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

(ASCEND) Re: What is carrier-class?

At 10:04 AM 11/9/97 +1300, Josh Bailey wrote:
>To address them both:
>
>1. This is leading edge technology.
>
>Our 7513 gave a lot of trouble during testing, refusing to swap RSPs
>properly on hotswap - with hardly any config in it. This is hundreds of
>thousands of dollars worth of equipment with minimal chance of any
>conflict with anything else - with one card configured up with an IP
>address. In the end, our reseller (a Gold partner) replaced the chassis. 
>
>What testing is done on the hardware to make sure it works before it
>ships? A carrier-class vendor would have caught this problem before it got
>to us.

Since I don't know the manufacturing process in detail, I can't
definitively answer your question.  However, it sounds like you had the bad
luck to get a DOA unit (or at least hardware-not-working-right-on-arrival).
 While Cisco does try to have 100% working units, we do have some that are
DOA.  The same is true of your carrier-class vendor, even if you haven't
experienced it yet.

>Brand-name PC vendors have been doing this for years. Burn the stuff in,
>*before* it ships. Even TV manufacturers do that, and I'm sure a lot more
>TVs get sold than Ascend Maxes...

I'm not sure this is true, but even if it is, think for a moment about the
problems it causes:

1)  The assembly line cannot automatically box product.
2)  A large amount of space has to be devoted to burn-in.  Cisco ships
thousands or tens-of-thousands of pieces of equipment every day.  Think of
how large a room you'd have to have to plug all that in.
3)  A large number of employees is needed to manage the burn-in process.
They have to plug-in, track, unplug and box up every unit.

Numbers 2 and 3 above are concerns, but not really your concern.  If it
needs to be done it could be done, but those two items will add significant
overhead to the manufacturing process.  

The biggest problem is that in a world so concerned with getting its
product now, the extra delay would be unacceptable.  Sure, you may be
willing to wait because you've done careful planning, but I'll bet the guy
next door to you is furious because he submitted his order yesterday and
wants the equipment onsite tomorrow.  And believe me, there's a lot more of
him than you.

If Cisco or Ascend came back to you and said - "we've created a special
class of customer, and you're part of it.  You'll now be assured of burn-in
and additional testing on all your products, but your lead time has moved
from 3 to 5 weeks," would that be acceptable?

>2. There are too many possible combinations...
>
>Fine. Which is why Telcos have the resources to pay for model
>environments. For instance, we have a model exchange, a model ATM node,
>and a model POP. So we *can* test for things in our environment in a
>consistant way before deploying them.

If you spend 10 million a year with Cisco or Ascend like you do with your
switch vendor, there may be some special consideration given.  However,
would you expect Cisco or Ascend to give you equipment for a model
environment if you spend $100,000?  How about $10,000?  Where do you draw
that line?  $10,000 is peanuts to you, but to that non-profit organization,
it's a huge expenditure.

>NEC give us a software release, which we test, find lots of bugs in,
>document them all, and send back to NEC - which they fix. NEC trust us -
>we trust them.
>
>The point is - Ascend/Cisco take the commodity paradigm too far. You guys
>treat all your customers like dummies. In fact, it's worse than that. A
>few months ago, an Ascend engineer came in to demonstrate a system for us.
>It didn't work. He declared it a bug. He had to reproduce the thing all
>over again with new debug before *Ascend* would acknowledge it! Seems
>Ascend doesn't trust even its own engineers. :(

As a Cisco SE, you're right, I can't submit a bug directly.  I have to go
through the same TAC engineers you do.  Is that because the TAC doesn't
trust me?  No, it's because it *can't* trust me.

Cisco has thousands of SE's.  How efficient would it be if I submit a bug,
and the DE (development engineer) spends a few hours only to discover that
I had misconfigured the piece of equipment and no bug existed?  Or that I
had forgotten to do a bug search and the bug was already documented and
being worked on by another engineer?  Now multiply that by thousands of
SE's.  Our DE's would never get any work done.  Sure, we'd submit lots of
valid bugs, but probably lots of invalid ones too.  

TAC on the other hand, goes through very specific training on what must be
done before a bug is submitted.  A bug search must be done, and the problem
must be reproducable in the lab (which most SE's in the field don't even
have a lab).  This results in the minimum number of "junked" bugs possible,
which means that DE can concentrate on fixing valid bugs and creating new
features, rather than chasing bugs which don't exist or are already
documented.

Now, that being said, I realize your frustration.  As a Cisco customer I
was a pretty darn knowledgable customer.  If I called with a problem, I
didn't really need someone asking me if the interface was turned on.  

Here again though, the problem is the different market.  A phone switch
vendor can pretty much bet that when the phone rings, there's a
phone-switch technician on the other end that probably has several years of
phone switch experience.  At Cisco and Ascend, we get everything from
network managers that can read packet traces straight from hex, to a guy
who heard he needed something called a router, so he called xyz mailorder
company and ordered one.  He's not sure what it is or what it does, but he
needs it to connect him to the Internet computer.  (And no, I'm not kidding)

Ok, simple solution, you say.  Just create two levels of TAC.  Advanced
customers get second-level, beginning customers get first-level.  Again,
how do we draw that line?  By the amount of money you spend?  That doesn't
seem quite fair.  By your opinion of yourself?  Perhaps a bit too subjective.

In Cisco's case, we created a special program for customers that are
highly-advanced, "carrier-class" customers.  It's called NSA.  Yes, it's
not a cheap program.  That's because it takes a lot of resources and
because your willigness to spend the money shows you are truly
carrier-class and willing to devote the resources on your end as well.
Most of our "carrier-class" customers have joined it and I'm told that once
they join, they don't leave because they like it so much. 


[several Ascend-specific issues I can't address deleted]

>1. Treating them not only with politeness but also if they have an IQ
>above 100. I was self-employed a couple of years ago. I won a lot of
>business because I didn't treat people who didn't understand what they
>were using as if they were stupid.

Agreed, 100%.  However, remember that the guy on the other end has no idea
whether you convert hex in your sleep or can barely spell router.  Help him
out.  Say, 

"Hi this is Josh.  I think I'm seeing a bug with PPP.  It happens when
xxxxx.  You can reproduce it every time by xxxxx.  If you'll tell me your
email address, I'll send you the output of show-tech and the output of
debug isdn events, debug ppp packet, debug ppp nego, debug isdn q931."

You tell a guy that, and it'll get his attention.

I myself fell into the trap of being lazy at times yet wanting TAC to treat
me like something special.  Don't call up and say:

"I've got this problem and I want a bug filed right now because I'm an
experienced user."

"Can you send me the config?"

"Uh, I don't have it.  I'll send it later."

"Can I access the box?"

"Uh, I haven't been there to put a modem on it yet, so no."

"Do you have debug output?"

"Uh no..."

Given that, this engineer is supposed to file a bug because of who you work
for?  I don't think that's realistic.

>2. Listen to what they say. There is a problem here. Perhaps it's config -
>perhaps it isn't. The attitude should be - fix it, don't point the finger.

Agreed.  However it *is* valid to eliminate older bugs as the source of the
problem by asking you to upgrade to the latest maintenance release.  If you
can't do it for some reason, fine.  We'll still work with you.  But it's
not a buck-passing - it eliminates one very likely cause.

>3. Acknowledge their concerns - "that's bad." They're ringing you because
>their business is going down the drain, and your box has something to do
>with it. Give them some confidence in the product - *show* them it works
>somewhere else.

Agreed here too.  However, I ask again, even though that TAC engineer knows
there are other customers running it in the same config you are, how's he
supposed to show you that without wasting time or violating another
customer's privacy (or wasting their time)?  I've been thinking about this
one and the only solution I can come up with is that you should ask your
local SE.  

Your local SE *does* have the time to ask around on some internal mailing
lists to try to find you other customers running the same config you are.
For various reasons, they might not always get a response, but most of the
time they should.

>4. Stop firefighting. I have worked at two ISPs where growth has been
>phenominal. Companies growing at 15%/month. Customers signing up
>1000/week. New staff being hired every other week.
>
>Yes, it is difficult to manage growth. Yes, it is difficult to cope with
>software versions. But not impossible and it's been done many times
>before. We cannot get away with "but you get that on the Internet" any
>longer.

Agreed.  And I wish customers could see all that goes on within Cisco to
try to manage that growth and to try to prevent firefighting mode.  Are we
always successful?  Absolutely not.  But we're always trying to improve.

Wade
---------------------------------------------------------------------------
Wade Williams                      "And the trees are all kept equal by
Systems Engineer                    hatchet, axe, and saw."
Cisco Systems, Inc.                       - N. Peart
Brentwood, TN                        
615-221-2918                             
wwilliam@cisco.com    
---------------------------------------------------------------------------
++ Ascend Users Mailing List ++
To unsubscribe:	send unsubscribe to ascend-users-request@bungi.com
To get FAQ'd:	<http://www.nealis.net/ascend/faq>
Follow-Ups: