Regulatory Issues

[stayouttadabunker]

"theft by omission of action"......hmm.....


You mean there's no such thing as "finder's keeper's" anymore? 

[brichter]

Finders keepers only works when you don't know who the lost property belongs to.You're probably not gonna get too far telling the jury you didn't know who the cash can belonged to...

[StatFreak]

Finders keepers only works when you don't know who the lost property belongs to.You're probably not gonna get too far telling the jury you didn't know who the cash can belonged to...

"But officer, I was just on my way to the gaming tables to bet the cash can... along with my bag of pot!"

[DealingwithNIGC542]

By God, that's right - I'd forgotten that part of the finders keepers rule, not that my older brother ever paid attention to it - lost more stuff to him than I care to remember... And, you're right - ain't no judge or jury buying the "how was I supposed to know??" arguement...

I should have clarified - how do you steal by omitting an action? I first heard of this at a casino where a cage cashier paid a customer $995.00 for a TITO redemption actually worth only $9.95. The tape review showed that the customer was aware of the cage cashiers' error (body language), but the customer said nothing (omission of action). The customer collected the cash, left a NICE tip and walked out. I don't have a clue if the customer was actually prosecuted (I was out of there just after that) but that's when I heard the term. I did Google the complete term but I didn't get anywhere with it - might have been a local infraction - in terms of (almost comical) oxymorons, it ranks right up there...

[doublediamonddlx]

I remember this one Indian casino I was in and when they scanned the TITO they had to place in a taped out square on the table...when the TITO got rung up it showed the amount on a LED board.  Cashier then had to place all money owed to the customer in a 2nd larger taped out square (including change) and you were not allowed to touch it until the dealer clapped and waved their hands...like in BJ.  Overhead camera taped the entire transaction and a 8' tall mounted camera captured you at the window.

Kinda funny how weve come along way from the cash/exchange cart rolling through the casino!

[DealingwithNIGC542]

Great mechanism for insuring proper payment(s) for TITO redemption - I wonder if they still do that. The transaction I was talking about - after the paper cleared the cage for that day and I saw the (almost) $1000.00 offage, I pulled that TITO from the cage paperwork. Turned out that the bar code was incomplete - a good TITO but the bar code couldn't be scanned. The cage cashier had to manually key the validation number into the system - tape review showed that the system did in fact return the $9.95 value - the cage cashier ignored that and paid the $995.00 anyway.. Needless to say, that cashiers' career came to an abrupt and ignominious end..

So, for something a little bit "different" - if you go back to the "dawn of time" on this topic (September, 2008), I originally had intended to start some dialogue in regards to "Regulatory issues"; I'd like to "test the waters" again (Knagl, I'd love to get your opinion on this). I have an interesting problem that I have been wrestling with for quite a while now and I'd like to post it BUT the problem is purely regulatory (although it does concern itself with the intricacies of "modern" games). There is a lot of history behind this problem; the reading is, ah, dry. Thing is, I think there are folks out there (LittlePaws) who are either currently trying to deal with this problem OR WILL be dealing with this problem in the VERY near future. Do I post here or what??

[jay]

You are most welcome to post, but the majority of us are not gaming professionals (home enthusiasts), most of the gaming professionals that do grace our shores tend to be more on the technical plane so the quality and quantity of your responses may be somewhat less than what you seek. We do however appreciate your participation on our forum and do invite your posts and views on all of our topics as you do appear to have some depth in areas that many of us do not.

Cheers

[DealingwithNIGC542]

That was very gracious, Jay - thank you. I've often marveled over (and cursed under my breath about) how gaming regulatory oversight is perceived by the public; the variety of opinions and perceptions is amazing. Perhaps we can clear some of that up... In so far as the "technical plane" is concerned; my experience has been that, where there is a technical problem, a regulation is close by. The folks that frequent this forum may be more technically-minded, but technicians know people who know people. I've also found the technically-minded to be curious sorts; the more they know, the more they want to know. I'm counting on that.

That said, I'll work up a "test post". It will cover ONLY the biggest regulatory problem I've had recently, just to let folks dip their toes in the water. Anyone wanting to get a jump on this should Google "25 C.F.R. 542.13 (h)(18)"; that's the portion of National Indian Gaming Commission Minimum Internal Control Standards (NIGC MICS) I'm fighting with right now. Enjoy!

Thanks again, Jay...

[Op-Bell]

I don't know if I qualify as a member of the public for these purposes, but I perceive gaming regulatory oversight as the morbid fear by government that someone, somewhere might be cheating them out of a nickel's tax.

[brichter]

I don't know if I qualify as a member of the public for these purposes, but I perceive gaming regulatory oversight as the morbid fear by government that someone, somewhere might be cheating them out of a nickel's tax.

How strange, that's my perception also, and I am a member of the -pub -lic. (They call me "Tater Salad" )

<edit> To clarify my position, I view most government regulatoy with the same view. The government seems to vacillate between telling the public what is good for them and telling the public how they need more money to tell the public what's good for them. And when things don't go right, it's always someone else's fault, and they need more money to fix it. I'm not saying all government is bad, but rather there are many parts of it that are.

/rant off

[DealingwithNIGC542]

Hmmm... I need to put more of a spotlight on my role in the regulatory scheme of things; try to dispel some notions:

I don't work for the Fed. Believe me, I don't have the kind of mind it takes to write regulatory language for the Federal government - those are some VERY SPECIAL people. 

I work for the casino, with local regulatory oversight (Gaming Commission). The role I play has to do with insuring integrity; protect the players, protect the properties, protect the industry. There are sizable chunks of Federal reg that stipulate the standards that must be met to (in the eyes of the government) insure overall integrity. I'm the guy (along with many, many others) that tries to interpret those regs and build a "mechanism" to gain compliance with said reg(s). Now, if reg calls for a mechanism to alleviate the governments' fear that tax nickles are slipping through the cracks, then I'm required to develop and build that mechanism. And, in reality, there are some regs in place out there that force properties to spend a dollar to find a nickle; it's the nature of the beast. That said, I believe that the OVERALL intent behind reg is good; if you run a regulated casino, you damn well better play by the rules - you get caught playing naughty, you'll pay DEARLY. That’s the way it should be. Consider how things might have been if there had been more focused, driven Federal oversight over investment banking and/or bond rating companies; I’m willing to bet the sub-prime mortgage fiasco would have been a flash in the pan rather than a meltdown.

That leads me to where I am now – I’m caught in a conundrum.  Casinos regulated under NIGC standards are required to evaluate the “performanceâ€� of the slots active on their respective floors to insure that the slots are operating within defined parameters (in regards to what the slots pay back to customers). Recent trends in the manufacturing,  testing and approval of slots are weakening casinos abilities to properly evaluate these newer games; those weaknesses must be addressed. More about that in a future post.

Anyway, that’s what I do and the reason I’m here, folks. Hope that helps….

[Op-Bell]

It's good to have a compliance guy here. Me, I'm from the other side of the table, the design side. I've had a lot of dealings with Regulation 14 (and Mr Mader's clones thereof). Federal regulations, that's something else again, originally related to stopping casino owners helping themselves to a handful of untaxed Franklins, and more recently to stop members of the public using casinos as a tax avoidance device. But as for the regulations being there to protect the players, don't make me laugh. I think the decision of the Nevada Gaming Commission in the case  last year where a player doubled down on ace-8 and drew a 2 laid that to rest. (For those who haven't heard about it, the casino (Terrible Herbst) claimed the player's total was 11, and the NGC upheld that.)

[jay]

Unless it is specifically written down and available for viewing I would be challenging that ruling in court. .... albiet I heard that Terrible is in financial trouble and being forced to see its properties... so blood from a stone comes to mind.

As a player I am more comfortable playing in a casino than a private game. First off I know that if the company is being traded publically that they wouldn't do anything to screw up their ability to remain on the market.
So when I go into a casino I feel that I do have a chance of winning on either the slot or table game. I think in absence of the oversight you would see public confidence errode.

[DealingwithNIGC542]

A DESIGN GUY – Flippin’ perfect!! I’m working on a post that you will (I hope) be able to comment on.

In so far as reg is concerned – there are (in tribal gaming) two sets of Federal reg to contend with; 25 CFR 542 (NIGC MICS) and 31 CFR 103 (“Title 31�). Your references to Federal regs created to stymie skimming and close tax evasion loopholes are part and parcel of Title 31 (aka The Bank Secrecy Act). Title 31 has no bearing on the issues I am contending with – those issues arose from contending with NIGC MICS.

In so far as regulation not protecting players; try this for additional context. The NIGC MICS were promulgated to provide minimum regulatory standards across the entire tribal gaming industry; the NIGC MICS were never intended to dictate to a tribe or casino management how they would conduct themselves during a dispute between a casino patron and the casino itself. The Fed leaves resolution of disputes where it belongs – with local regulatory authorities. Insofar as the decision made by the Nevada Gaming Commission? No opinion without having read the case materials. Do regulators screw up? You bet. Thing is that many millions of hands of blackjack are dealt daily – I’d bet the country as a whole sees tens of millions of game starts on slots each day – do the math…

And, Jay - You hit a viseral point - if public confidence that casinos (Vegas or otherwise) operate "fairly" was to erode, the industry would slowly crumble. All tiers of regulatory authority safeguard that confidence.

[Op-Bell]

First off I know that if the company is being traded publically that they wouldn't do anything to screw up their ability to remain on the market.
... I think in absence of the oversight you would see public confidence erode.

I would cite "Scarne on Gambling" as the authority here. As long as an establishment is permanent, it needs to protect its reputation; and given the house edge, there is no incentive to cheat a player. Nobody ever accused the old Vegas managements of cheating at play; by most accounts, players got a much better deal overall before the corporations took over.

Of course, without oversight there's no control over the house edge, but competition takes care of that. People won't play 80% games if there are 90% games next door (well, apart from Megabucks and WoF). Years ago, Harold's Club in Reno set some of its slots to 96% by accident, and their drop increased so dramatically from increased play that every other Nevada casino followed suit.

[stayouttadabunker]

That's called..."going with the flow"...

[kforeman]

That was very gracious, Jay - thank you. I've often marveled over (and cursed under my breath about) how gaming regulatory oversight is perceived by the public; the variety of opinions and perceptions is amazing. Perhaps we can clear some of that up... In so far as the "technical plane" is concerned; my experience has been that, where there is a technical problem, a regulation is close by. The folks that frequent this forum may be more technically-minded, but technicians know people who know people. I've also found the technically-minded to be curious sorts; the more they know, the more they want to know. I'm counting on that.

That said, I'll work up a "test post". It will cover ONLY the biggest regulatory problem I've had recently, just to let folks dip their toes in the water. Anyone wanting to get a jump on this should Google "25 C.F.R. 542.13 (h)(18)"; that's the portion of National Indian Gaming Commission Minimum Internal Control Standards (NIGC MICS) I'm fighting with right now. Enjoy!

Thanks again, Jay...

hello there mister regulator...i'm a lead tech at my casino and have been techin for nearly 7 years now so i have been working closely with our regulators for quite some time.  i'm the first person to admit that i definitely don't know all of the NIGC MICS word for word, but i do feel that it is my resposibility to make sure none of my techs are in violation at any time, therefore i try to at least have a working knowledge of the rules we must adhere to.
i read the section of the MICS you are working with right now and am totally willing to offer my views and or ideas that may help you.  as the others have said, we welcome the regulatory aspect of gaming to our online community. 

[DealingwithNIGC542]

So – here it is – the promised post in regards to the NIGC reg(s) I’ve been contending with. Some background: I started this chase back in June of 2008, after attending a seminar on evaluating slot hold/payback performance. Ken Moberley taught the seminar; Mr. Moberley is an educator with the University of Nevada and a former executive with IGT. I did some additional research and contacted a gentleman by the name of Don Caitlin; Mr. Caitlin is a retired educator and writes articles about gaming, with a particular slant on slot machines. Mr. Caitlin put me in contact with a Dr. Hannum, a PhD in mathematics at the University of Colorado, Denver. Randy Fromm, publisher of Slot Tech magazine was copied on most of these mails, along with Mr. Jason Buckingham, Internal Auditor for the casino where I work. I’ve included excerpts of the e-mails that I felt were relevant; I apologize in advance for the length. I wanted to be sure the “jist� of the problem is clear.

July, 2008

Hello Mr. Catlin:
 
Recently, I have taken on the responsibility of investigating slot machine actual to theoretical hold variances; I've been spending a lot of time educating myself on slot mathematics. I found your name while doing research; two articles published on the Casino City Times web-site in reference to the Inge Telnais patent and random number generators (amongst many others). I'd like to make an observation and ask your opinion on a question:
 
The observation: I've been pulling PAR sheets on games from various manufacturers as part of my variance work and I notice that some of the games we're running on our floor have huge cycles; as an example, we recently installed a game has a cycle of 943,800,000 combinations. My arithmetic (based on 60 days of accumulated game starts) indicates that this game will (theoretically) complete 1 cycle sometime in July of the year 3341. In addition, some of the reading I've done (and opinions I've heard) indicates that games with cycles in the billions (and even hundreds of billions with server-based games) will soon become common place.
 
The question: The portion of Federal reg that governs investigation of actual to theoretical hold variances for tribal gaming {25 CFR 542.13 (h) (18)} requires we provide written explanation(s) for any slot that has accumulated 100,000 (one hundred thousand) game starts and is operating outside of a +/- 3 (three) percent variance window, actual against theoretical hold. If manufacturers will not provide basic diagnostic tools; with Federal regs falling far behind industry technology (look at machines with 100,000 game starts against a cycle of 943.8 million?); with cycles that will never (theoretically) complete, what other tools can we use to PROPERLY evaluate the performance of these games?
 
Thanks for taking the time if you can, Mr Catlin. I appreciate any advice or opinion you are willing to offer.
 
July 2008

From Mr. Caitlin
 
I really don't know what one can say about a machine that is operating at 3% off its theoretical mean other than to say that the random number generator is flawed.  That doesn't seem credible, though, since I know that slot manufacturers go to great pains to make sure that their RNGs really do simulate randomness.  Sorry that I can't be of more help but I just don't know what to tell you.  You might try Robert Hannum at Denver University; his work deals with both statistics and gambling law.
 
One comment.  Slot machines spit out random numbers hundreds of times each second.  If a slot machine has an RNG with a cycle of 943,000,000 and generates 1000 random numbers per second it would cycle through every number in about 11 days.
 
Don Catlin 

August 2008

Hello Dr. Hannum
 
We are a small tribally-owned NIGC classified Class III gaming facility. Don Catlin suggested I contact you to solicit your opinion(s) in regards to some issues we are contending with and to invite you to join in some dialogue regarding the direction on the industry in general.
 
Recently, I have taken on the responsibility of investigating slot machine actual to theoretical hold variances; I've been spending a lot of time educating myself on slot mathematics. I'd like to make a couple of observations and ask your opinion on a couple of questions:
 
The observations:
 
1.) I've been pulling PAR sheets on games from various manufacturers as part of my variance work and I notice that some of the games we're running on our floor have huge cycles; as an example, we recently installed a game with a cycle of 14,505,961,008 combinations. We have another game with a cycle of 34,359,738,768 combinations. I understand that server based games have cycles in the 300,000,000,000 combination range.
 
2.) I've noted that manufacturers are becoming very stingy when providing performance evaluation tools (90 or 95% confidence tables, standard deviation, volatility indexes, etc.).   
 
The question(s):
 
1.) It appears (to me) that we are caught up in a "circular firing squad". The industry continues to submit games with monstrous cycles and they continue to be approved; casinos must purchase these games to remain competitive. Federal regulations/regulators will not budge on compliance with existing regs despite this disturbing trend nor are they attempting to control it. Manufacturers withhold performance evaluation tools from thier customers - In your opinion, what is driving this trend and where did it come from?
 
2.) The portion of Federal reg that governs investigation of actual to theoretical hold variances for tribal gaming {25 CFR 542.13 (h) (18)} requires we provide written explanation(s) for any slot that has accumulated 100,000 (one hundred thousand) game starts and is operating outside of a +/- 3 (three) percent variance window, actual against theoretical hold. If manufacturers will not provide basic diagnostic tools; with Federal regs falling far behind industry technology (look at machines at 100,000 games starts with game cycles in the billions?); what other tools can we use to PROPERLY evaluate the performance of these games?
 
I/We appreciate any opinion or advice you care to offer, sir. Thank you for your time.

August 2008

From Dr. Hannum

I’m not sure how helpful my thoughts will be (and I’m sure you are aware of much of this), but for what it’s worth, here goes.

1)      With outcomes generated by random numbers, huge cycles are possible. This gives game designers more flexibility in creating games and/or payout schedules (as well as larger jackpots). I view this as a good thing.
2)      In my experience manufacturers have always been stingy in providing certain details, including performance evaluation tools, about their games. I view this as a bad thing.
3)      In my experience gaming regulators seem to be extremely slow in implementing or modifying gaming regs to keep up with technology. Again, a bad thing. I would hesitate to speculate as to why this is but would certainly endorse this changing.
4)      Quick calculations suggest that a machine with about a 95% payback and standard deviation of 5% will should have 90% limits of roughly 92.4% to 97.6% after 100,000 pulls. The analogous limits for a machine with the same hold but a 10% SD are 89.8% to 100.2%; with a SD of 20% it’s 84.6% to 105.4%. The point is, it would not be surprising for some properly operating machines to fall outside the +/-3% window after 100,000 pulls. I’d have to give more thought as to exactly how the actual to theoretical hold evaluation should be done, but it seems to me it should be somehow tied to the machine standard deviation (variance); right now it isn’t. In addition, I would tend to increase the number from 100,000 (though might be implicitly accomplished with a revision of the evaluation that is based on the SD). Finally, I never liked the industry standard of a 90% confidence level for slot volatility analysis, inherent in the volatility index using a 1.65 z-score; in my (strong) opinion it should be at least 95% (conventional in most statistical settings).

As I said, for what it’s worth…

Cheers,
Bob

My reply, September 2008

Hello Dr. Hannum,
 
At the outset, let me express my/our gratitude that you took the time to respond to us.
 
In so far as your responses to the questions I posed:
 
RE: 1.) Further research on my part did disclose that your view is right on the money (no pun intended). Designing games with huge cycles does offer programmers tremendous flexibility. While internal policy prevents me from playing on our floor, I have taken a closer look at some of these games and I am impressed at the level of "customer interaction" they offer ("nested" bonus rounds, "players choice" bonuses, etc.). I'm gaining a better understanding of the need to manipulate pay distribution(s) and bonuses. That said, I'm still not convinced that these huge cycles are a "good thing"; please read on.
 
RE: 2.) I concur with your view that lack of performance evaluation tools is a bad thing. If I might go a bit further:
 
It is clear that game developers are capitalizing on rapid changes in technology. If I understand correctly, the "next generation" of server-based games will have the capacity to run hundreds (and potentially thousands) of games; these games will have cycles in the hundreds of billions. While it seems clear that regulatory bodies are comfortable with these advances, my feeling is that these bodies have lost sight of an important point - why the regulatory bodies exist to begin with. The research I've done up to this point tells me that the intent behind many of the regs we must contend with are represented by three basic tenets:
 
       Regulation(s) are designed to protect the properties, protect the players and protect the integrity of the industry.
 
Then I ponder the enormous complexity of software development; I invite you to follow the link below. This short power point presentation reveals some of the pitfalls of software development and engineering. It was created by Jyrki Nummenmaa (Head of the Department of Computer Sciences, University of Tampere, Finland).
 
http://www.cs.uta.fi/~jyrki/se03/motivation.ppt
 
As Dr. Nummenmaa points out, some of the examples provided are pretty humorous, but the others are very serious. I understand that the parallels are a bit extreme, but I think you can see my point. Purchasers/end users of these complex games are completely at the mercy of the game developers to write "good code"; properties have no diagnostic tools "on a par" with the complexity of the game(s) to accurately evaluate their performance.
 
As an example, look at this scenario (and please bear with me; I'm a layman wading into DEEP water):
 
An imaginary casino has a $0.25 denom game running with a cycle of 34 billion combinations. Our hypothetical casino sees 150,000 game starts a month on this particular game (single coin wager, that's $37,500.00 per month in coin-in.) At a 93% theoretical payback, our casino should be seeing around $2,625.00 in theoretical net win each month. At 100,000 game starts (0.000294% of a cycle complete) let's say that the game is operating at 89% actual payback (or $4,125.00 in actual net win). That actual payback percentage is 4 percentage points away from theoretical and, based on the standard deviation tables I've seen, well within range at 100,000 game starts. This is what bothers me: 
 
Our casino is seeing a 57% increase in net win on this game and it is just BARELY operating outside the +/- 3% variance window provided by (NIGC) reg. Then calculate INSIDE the variance window - using the same scenario, with a +/- 2.5% variance actual against theoretical hold (95.5% to 90.50% theoretical payback), our game could be generating anywhere between $1,687.50 and $3,562.50 in net win each month. 
 
Now, my understanding is that theoretical payback percentages are based on a "lifetime span". Most of the standard deviation tables I've worked with are calculated out to either 10,000,000 or 20,000,000 game starts; I'm sure those game start numbers are reasonable in a volume operation. What I can't get my head around is the relevance of those game start numbers against a game with 34 billion combinations in a cycle. At 10,000,000 game starts, that's 0.0294% of a cycle complete. At 20,000,000 game starts, that's 0.1% of a cycle complete. How can we rely on a 90% confidence table (or even a 95% confidence table) calculated to 10 or 20,000,000 game starts when those game start numbers are (in my view) inconsequential when compared to the overall game cycle? In my view, given the complexity of the game(s) and the "bluntness" of the diagnostic tools currently in use, casinos could be running games on their floors for DECADES (or even longer) before a problem MIGHT show up. Hundreds of millions of dollars could be "exchanging hands" when it shouldn't be and NO ONE WOULD EVER KNOW. That concept brings me back around to the previously mentioned regulatory bodies and those three basic tenets.
 
RE: 3.) I often ponder the lack of Federal-level "spryness" in regards to changes in regulation(s). Current events (the pending Fed takeover of Fannie Mae and Freddie Mac and the whole mortgage meltdown appears to be a good example) would seem to support the theory that something catastrophic has to occur before any significant change takes place. I wonder what kind of a catastrophe would have to take place in the gaming industry to get the various regulatory bodies to take a closer look at the current trends in game development, approval procedures and performance evaluation. Food for thought...
 
RE: 4.) In conducting research to attempt to find additional evaluation tools, I stumbled across the link below. I/we would like to solicit your opinion about this utility and ask you if you are aware of any other utilities of a like nature. This utility is still a bit  blunt, but it is better than what we have now.
 
http://downloads.zdnet.com/abstract.aspx?docid=342606
 
Dr. Hannum, again let me express my/our thanks for any opinion(s) you choose to offer. Good day.
 
NOTE: I did not receive a response back from Dr. Hannum – at this point in time, he was dealing with a “serious and on-going family emergency� (his words). I didn’t feel it appropriate to press him.

September 2008

This mail from the Internal Auditor at my casino:

Hello to all involved in this post and a big thank you to all involved,
What I see as the major problem, from a regulatory perspective, is that you have these huge cycles that players seem to love and no tools available to determine acceptable deviation from theoretical hold percentage.  I.e. game A is at 150,000 game starts and is currently holding 110% and theoretically could be at 110% for years before it should statistically come within an acceptable (profitable) range. 
I wonder how volatility is kept in check with these games?  Is it sheer volume, spread over numerous machines that will keep the risk of volatility within an acceptable range?

One important thing to remember is that we hold the power, as regulators, to allow or prohibit these types of machines on our gaming floors.  We should be demanding manufactures to provide us with the tools we need to measure acceptable deviation prior to licensing the gaming machines.  As many of you know, our compacts provide little regulatory guidance in today’s gaming world, the NIGC is also a few steps behind and seems to be more focused on creating work for gaming operations that provide little to no value, so it is up to us. 

So here comes the real question,
How do we stay competitive in today’s market while maintaining control levels that protect the integrity of out industry?

September 2008
My response to our Internal Auditor

Thanks for chiming in, Jason. The bit about possible prohibition of certain games is just what the conversation needed. I wanted to get something straight, though - You and I chatted about the lack of diagnostic tools and I came away from that conversation under the impression that GLI could provide us with deviation tables and the like, upon request. We pulled the letters of certification for the games in question, but they proved to be of no help. Can we still use GLI as a resource? Do I/we need to build some kind of formal request system, etc??

NOTE: I did not receive a response from Mr. Buckingham. It was at this point that Mr. Buckingham ah, recused himself from further comment. Mr. Buckingham left the employ of the casino shortly after this to take a position with another company.

This issue did not die after this last mail. In March of 2009, one of our Gaming Commissioners attended a conference hosted by Gaming Laboratories International, one of the leading game testing and approval companies on the planet (GLI does much of the testing for games for the Nevada jurisdiction, as an example). One of the GLI moderators forwarded this set of mails to two senior staff members of GLI at the end of March, 2009. We have received no response from them as of the date of this posting. Also, I checked both of the links included; as of this post, they were both still active. I invite you to check them out - they really help with the context of the problem (as I see it).

Time to stop and let those that have the time digest this post and ponder the implications. 

[jay]

MY observations and comments:

With a high probability (100,000 starts seems high to me) of a 3% variance from the published standard deviation would imply that you would not want to have any games with a > 96% payback as it would appear that this variance could consistently imply financial losses to the establishment.

If the principal role of the regulator is to protect the casino, then you would want to take appropriate measures.

As an example ... I would calculate the average drop and subsequently determine my cost of operations as a percentage. The derivative would then allow me to ensure my hold % was greater than this number. Personally I would be inclined to add in 15 points for a decent profit margin. However knowing that this alone would put me below 85% I would need to figure out what my amortised investment cost was vs drop and ensure that I was creating value through ROI even if it only worked out to be 1 or 2 points against the drop.

As an IT professional I ensure system (and subsequent financial systems) stability by rebooting my servers once a month. Some operating systems are more stable than others but I am inclined to believe that most are complex enough to potentially have memory leaks. I defer to your referenced PPT on software engineering in this regard. The reboot frequency ensures that everything remains within operational norms. Extrapolating that to the casino, and being somewhat naive of gaming law would it be possible to put in a procedure where you do a game clear every 3 months. This would perhaps ensure that the varience would never exceed the 3%, and as such never reach the threshold of the reporting requirement. IE In SOX terms the PROCEDURE is the CONTROL that ensures COMPLIANCE with the law.

Being a novice slot operator in a home enviornment the IGT S+ PAR sheets are based on 10mm spins. This sheet only provides me the theoretical payback % to work with.  I would think that standard deviation norms of sucha a cycle would easily exceed 3%. However with the slot accounting available from the various internal counters you should be able to segment some of the values and nomalize these. The remaining samples which would then would represent out of norm charastics would be statistically insignificant. This might not work for all variences but it would help explain a good portion of them.

Finally if the slot has not completed its cycle would that not be an explanation enough as to why the slot is out of compliance.

[brichter]

As an IT professional I ensure system (and subsequent financial systems) stability by rebooting my servers once a month. Some operating systems are more stable than others but I am inclined to believe that most are complex enough to potentially have memory leaks. I defer to your referenced PPT on software engineering in this regard. The reboot frequency ensures that everything remains within operational norms.

That's a good strategy for the service provider end of the business, Jay. As someone from the QA side of the business, I try to ensure that my UUTs (Units Under Test) never get rebooted, explicitly to find the isssus you attempt to avoid with your periodic reboots. Proactive action like this on both sides of the coin ensures customer staisfaction.

Shortly after we acquired one company several years ago (towards the tail end of one of their releases), we had customers find no less than 4 memory leaks in one component of the product within a week of the release. I was on the post-mortem team that assessed why this occurred, and in our research we discovered that the test team in question was rebooting their test units after every category of test cases was completed, thereby ensuring they would never find any memory leak related issues in the product.

[DealingwithNIGC542]

I’ll frame my response in reverse order, Jay.

The “connectionâ€� between the completion of a game cycle and the performance of the game is purely theoretical. While the application of common sense would tell you that, as a game approaches the “endâ€� of a cycle, the payouts from the game should also be nearing completion. That common sense application COULD work if the game program was based on a bingo or pull-tab style game – what the NIGC calls a “Class IIâ€� game. With a Class II game, each time a game start outcome is determined, whatever combination of symbols that resulted from that game start is “discardedâ€�; it will never occur again. With the games I’m dealing with (NIGC “Class IIIâ€� games), once a game start outcome is determined, that combination of symbols “goes back in potâ€�; that outcome could be generated again. These newer games will NEVER complete a cycle. Besides, the whole idea of evaluating actual to theoretical hold is to guarantee (as much as possible) that a particular floor is “fairâ€�; patrons are being paid back what they should be paid  and the casino isn’t making more than is should.

You’re lucky to have the PAR sheet with a standard deviation table, Jay; I’m willing to bet, if you look, the sheets will have a volatility index, etc. The PAR sheets for the games I’m working with DON’T HAVE THEM. I/we can’t even determine if some of these games are theoretically operating within “normal� bounds. NOTE: If you check the standard deviation table on your IGT game, there probably will be a statement that says something like “90% confidence�. In essence, what that means is the spread in the theoretical payback on the game (based on XX number of game starts) is AT BEST 90% reliable.

Casino operators LOATHE the idea of booting their servers; the floor is “shut down�. In addition, (and I’ve seen this happen), there is a risk that the servers won’t come back up. Casinos do reboot servers, of course, but only in circumstances that really require it. Clearing RAM on the slots won’t work, either. There are regulatory issues and besides, the game goes back to Square 1 – letting game starts accumulate (on older games) was (and still is) effective in gauging performance (actual to theoretical hold). Thing is, those older games had cycles of a million combinations or less; a game played heavily could “cycle� in a couple of years (or less). These new games will NEVER cycle even once.

In so far as theoretical hold percentages and profitability are concerned; depending on jurisdiction, there are very strict rules in regards to what is allowed on a casino floor (in terms of hold percentages). Your opinion makes perfect sense from the profitability standpoint, but your casino would be out of business. Competition would kick your butt, you’d be in bankruptcy court or (maybe) in jail…

Lastly, I’m NOT a regulator – I don’t promulgate ANY of these regs. I work on the casino side of the scheme of things to create mechanisms to satisfy reg. That’s the whole idea of doing this; I know there are other people out there with the same problem(s) I have – I’m hoping to find them and get their ideas.

By the way, is SOX a reference to Sarbaines-Oxley??

Hope that helps, Jay…

P.S., brichter – you’re scaring the hell out of me, man. What you’re talking about is EXACTLY the kind of thing that could “get by� – From the time your customers picked up on the problem – how long until the post mortem was completed?

[jay]

SOX = Sarbaines-Oxley

[brichter]

So, I don't believe Jay was suggesting that casinos reboot servers or machines, he was addressing points made in the presentation you linked as to why software fails. His business can schedule downtime and there are almost always failover or redundancy features built into software products marketed to enterprise-class businesses so customers don't feel the effects of planned maintenance. This is not always the case, and it seems from your post that this type of planning for outages has been not addressed by the gaming industry yet.

As far as the post-mortem goes, we released the software on a Friday as is customary, so engineers at the customer sites can upgrade over the weekend. This gives them time to recover from any issues they may find. Recovery is anything that must happen in order to get the sytem back up, including reverting to the older version of software.

Since these were memory leaks, they don't manifest themselves until the system has been in operation for some time. The first issue was seeen on Monday morning (local time for us) from a large cutsomer deployment in Europe whose system has been up all day. We assigned a developer to work with a support engineer to root-cause the issue, and by Tuesday night, we had the other 3 issues reported. Note that these were not issue with the same code module, but were different problems. At this time, we realized we had multiple pattern failures, and the PM team was assembled. Wednesday and Thursday we gathered info and presented the findings Friday morning, and had new testing procedures in place the following Monday. That was not a fun week (or weekend) for me.

There were also changes in process on the development side to enforce in-depth code reviews by peer developers. Code reviews were a requirement before the incident, but were not enforced, and as a result were either just being pencil-whipped (signoff without proper review of the code) or not being performed at all.

The 4 issues were resolved in parallel and patches issues by Thursday evening local time, and equipment was being rebooted nightly on customer sites until the patches were released.

[Op-Bell]

#1, Class 3 games don't have a "cycle". That's just a statistician's sophistry to work out theoretical outcomes. Class 3 games are truly random, and in a truly random system, every possible combination of results is possible - that includes 100,000 losing plays in a row, and 100,000 jackpots in a row. The statistical probability of such an outcome is very small indeed, but it exists, just as the probability exists in quantum theory that all the atoms in a girl's clothing will simultaneously jump three feet to the left (Hitchhiker's Guide to the Galaxy). In games with a very large "cycle" it's entirely possible that some results will never show up in the lifetime of the game.

#2, in reality, however, the numbers are not as big as you think. Take a 5 reel video with 256 stops per reel. The total number of combinations (the "cycle") is 2^40, about 10^12, one trillion. However, there are not 10^12 different game outcomes, since many of them are the same. The more you increase the number of stops for the same number of outcomes, the more the outcomes tend towards the statistical mean. If you read your tribal compact you'll almost certainly find a clause that says the long odds on any jackpot can be no greater than 16.7 million to 1 (2^24), so in a trillion plays, at least 65,000 of them should be the jackpot. Jackpots should in any case be excluded from this sort of compliance test. That is, if the game is 95% and 5% goes to the jackpot, evaluate it as a 90% game and don't count any jackpot that hits during the test.

#3, playing multiple lines reduces the volatility (variance, whatever). To get results closer to theoretical, play maximum lines. To get the largest chance of variance, play single lines. The most dramatic example of this is in Action Gaming's 100-play poker. Jeez, what a yawn. Play $5 a hand, win $4.85 (+/- 25c) every time. Even hitting a couple of royals in the hundred hands doesn't make much difference.

#4, Put not your faith in GLI, or any other test lab. Maybe in the past they did full evaluations of game software, but these days, especially with the move to use third-party code and proprietary operating systems like Windows for which there's no source code, it is far beyond their capability. All they can do is test the game to see if it complies with the manufacturer's own specifications, pretty much the same as you can. Not even the original designers can know all the twists, quirks and vulnerabilities. A smart programmer could put a gaffe in there with virtually no chance of anyone spotting it - and who's to say they haven't already?

[DealingwithNIGC542]

Jeez, Op-Bell - between you and brichter, I'm gonna stroke out.

Clarify something for me:

#1, Class 3 games don't have a "cycle". That's just a statistician's sophistry to work out theoretical outcomes. Class 3 games are truly random, and in a truly random system, every possible combination of results is possible - that includes 100,000 losing plays in a row, and 100,000 jackpots in a row. The statistical probability of such an outcome is very small indeed, but it exists, just as the probability exists in quantum theory that all the atoms in a girl's clothing will simultaneously jump three feet to the left (Hitchhiker's Guide to the Galaxy). In games with a very large "cycle" it's entirely possible that some results will never show up in the lifetime of the game.

All of the training I've taken up to this point says there ARE (theoretical) cycles in Class III games. The first (elementary) training used a very familiar, very popular 3 reel, 64 stop game - cycle 262,144. NIGC classifies this game as Class III and the evaluation process I was taught translated very nicely to other games of this type (we're still running many of these games). What you're saying flies in the face of that. Are you saying that the time-tested, many times reviewed process I was taught is nothing more than smoke and mirrors?

Between the two of you (now that I'm losing my mind) - can you thumbnail the architecture of corporate software development? What I know about this you could write on the back of a matchbook in grease pencil. Sounds as though game developers (and testers) could have the same kind(s) of procedural vulnerabilities that brichter's company had.. 

All of these games that have no eval tables or volatility indexes - these are ALL high hit frequency, low volatility games. The bit about not trusting the testing labs - Op-Bel - you've got it. I didn't want to come right out and talk smack about them BUT I can't understand why we can't get ANYONE to talk to us about this. The whole "confidence in the industry" thing rides on this. What ARE the game developers and testing labs doing?? Why can't we get any answers? Who the hell do we have to SCREAM at to get something moving on this?

Thank you, gentlemen - Excuse me while I open the bottom right-hand drawer of my desk, retreive my (not supposed to there) bottle of sour mash and pour myself a couple of fingers - Jeez-Louise...


P.S., brichter - "pencil-whipped" is now part of my regular vocabulary - give folks arouind here something to ponder...