1 1 BIDDERS CONFERENCE 2 SAN FRANCISCO, CALIFORNIA, JULY 30, 1999 - 1:30 P.M. 3 * * * * * 4 MR. LAKRITZ: Good afternoon, my name is Jonathon 5 Lakritz. I work in the Telecommunications Division of 6 the California PUC. 7 And we are conducting our bidders conference 8 related to the RFP for a test administrator and a 9 technical advisor. At this bidders conference today we 10 will answer any questions that bidders, potential 11 vendors, may have about the requirements in the RFP, or 12 in general about the OSS testing process that the 13 Commission is undertaking. 14 As a preliminary matter, I just wanted to 15 review the schedule that we are going to be using in 16 terms of selecting a bidder, or I'm sorry, a vendor for 17 the technical administrator and for the, sorry, for the 18 test administrator and technical advisor. You have to 19 pardon me, I have a cold, I'm a little foggy today. As 20 you are aware, the bids are due on August 10th, and the 21 interviews are going to be held August 18 through the 22 23rd. We'll be contacting those people that respond to 23 the RFP individually to set up interview dates. We 24 expect to select a vendor shortly after the interviews, 25 and that the vendor would commence work on September 26 1st. 27 In the original RFP dated June 30th, the 28 Commission intended that the test execution phase would PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 2 1 begin by September 1st, 1999. Obviously, since we're 2 not going to be picking a vendor until approximately 3 September 1st, that's no longer realistic. We're not 4 setting specific dates for the test to start executing 5 at this point in time, rather, we are requesting the 6 vendors to keep in mind our original time schedule, 7 which gave them about four to six weeks to be ready from 8 the point at which they would start to when the test 9 execution would commence. And we're asking that 10 vendors, as they respond in their proposals, indicate 11 whether they feel that's a realistic time frame, and if 12 it's not, that they would indicate what they would 13 consider to be a realistic time frame for that process. 14 As we learned when we selected a test generator, that 15 the time frames were slightly different than what the 16 Commission had originally envisioned. It was sort of 17 shooting in the dark when we were setting up time 18 estimates with you. We felt we needed to put something 19 up for people to respond back to. 20 I should tell you that there is going to be an 21 update to the RFP on Monday. The third update, I 22 apologize. At that point, we will be asking people to 23 include in the proposals a project plan and a brief 24 description of how the vendor would accomplish the tasks 25 outlined in Section 5.2.4 in the Master Test Plan, the 26 draft that was issued on June 24th. We are asking for 27 this information to help us better understand the timing 28 of how the project would unfold and to get a better PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 3 1 sense of how the vendor would accomplish the individual 2 tasks listed in Section 5.2.4. 3 Lastly, and you may be aware that we contacted 4 some, we contacted all of the test generators to clarify 5 a potential conflict of interest problem, and that had 6 to do with the same entity being both the test generator 7 and the test administrator and the technical advisor. 8 And based upon the appearance that there may be a 9 conflict when escalation clauses that are contained in 10 the draft of the Master Test Plan are put into place, we 11 have instituted a subsequent change that the vendor that 12 is selected to be the test generator will not be the 13 same vendor that's going to be selected to be the test 14 administrator. So, for instance, if we selected vendor 15 A to be the test generator, vendor A can bid to be the 16 test administrator, but it will not be selected or 17 awarded the contract to be the test administrator. 18 Similarly, we have the same types of concerns 19 about subcontracting, so let's say we went through the 20 process and selected vendor A to be the test generator 21 and vendor B to be the test administrator. If vendor B 22 plans on subcontracting with vendor A to do a portion of 23 their work, we would probably not allow that. And so, 24 as a result, one of the questions that we will be asking 25 during our interviews will be what subcontractors 26 vendors plan on using, if any. 27 I realize I just presented a lot of 28 information. I want to take questions in just a moment, PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 4 1 but before we do that, we're having this all 2 transcribed. It will be available on Monday. We will 3 E-mail copies to people. There's a sign-up list going 4 around the room, so we ask you to sign up on it. But as 5 a result, when you ask questions, if you can please 6 identify who you are and what company you're from before 7 you ask the question, and it will make it easier for 8 both ourselves and the court reporter. 9 Anyone have any questions about what I just 10 presented? I think at this point -- I'm sorry. 11 MR. DRAGER: If it's appropriate -- 12 It's John Drager from Telcordia Technologies. 13 You say the new thing we will look for -- the 14 RFP amendment we will look for will ask us to respond 15 to, I think you said the word "how" the test 16 administrator will perform its role in Section 5.2.4. I 17 presume you mean schedule related, because you already 18 were asking this stuff before. That's my question, what 19 do you mean by the term "how"? 20 MR. LAKRITZ: I think we're looking for a one- or 21 two-paragraph description for each one of the rules 22 describing what the key methodology is that the person 23 would use, or the techniques, or maybe some of the 24 sub steps to get an idea to see how the vendor would, 25 for instance, you know, examine the scalability of the 26 electric interfaces and the scalability of the local 27 service center and the local operation center. 28 Go ahead, please. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 5 1 MR. DORRIAN: Yes, Michael Dorrian, Lockheed 2 Martin. 3 Just clarification on the, on some of the time 4 lines obviously depending upon the availability of the 5 test transaction generator. Do you have an approximate 6 time line as when that might be available, the test 7 transaction generator, the testing bid? 8 MR. LAKRITZ: What the vendors generally have told 9 us -- since we haven't made public who the vendor is, I 10 can't give you an exact time line, but generally what 11 the vendors have told us is that they expect that they 12 can have an interface up in about six to eight weeks 13 from the date they start construction of the interface. 14 That is dependent upon receiving certain information 15 from the test administrator, such as the test cases and 16 there would be trial accounts to do some of their 17 initial work with. 18 MR. DORRIAN: Okay. So they can't start it until 19 the test administrator has been selected then? 20 MR. LAKRITZ: No, most of them have indicated that 21 they might be able to do some of the work -- there's 22 approximately three weeks' worth of training that they 23 need to undertake, and some of them have indicated that 24 they would be able to undertake that training prior to 25 the test administrator being in place. 26 MR. DORRIAN: Okay. 27 MR. LAKRITZ: To be honest, the final answer to 28 that question would be dependent upon when the test PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 6 1 administrator and the technical, the test generator are 2 able to sit down and have full discussions about what's 3 expected from each party. 4 MS. ALEXANDER: I have a question. Bonnie 5 Alexander, Network Intelligence. I'm sure she can hear 6 me. 7 Technical advisor is different from the test 8 administrator. As a technical advisor, is that the 9 Commission? 10 MR. LAKRITZ: No. In the July 6th update to our 11 June 30th RFP, we are asking entities to jointly bid on 12 both the test administrator and the technical advisor. 13 MS. ALEXANDER: I saw that, and I was wondering, 14 because I have the June 30th, and I'm wondering if that 15 was misread or what. You want a technical advisor who 16 is the test administrator? 17 MR. LAKRITZ: Right. One entity performing both 18 roles. In the Master Test Plan there are three entities 19 that are listed, but we see one entity performing both 20 the technical advisor and the test administrator roles. 21 MS. ALEXANDER: Right. 22 And in the RFP is it broken down for the 23 technical advisor information what role that technical 24 advisor would play? 25 MR. LAKRITZ: Yes, in the July 6th update, which is 26 posted on our Commission Web site. 27 MS. ALEXANDER: Okay. I have the June 28th version 28 of that. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 7 1 MR. LAKRITZ: There's a later, a letter dated June 2 30th, which described basically the test administrator's 3 responsibilities, and then there's another letter dated 4 July 6th which describes the technical administrator's 5 responsibilities. I'm going off the versions that are 6 on our Web site, because those are what we consider to 7 be the final versions. 8 MS. ALEXANDER: Thank you. 9 MR. LAKRITZ: Sure. 10 If anyone has any additional questions at this 11 point in time, I'd be glad to attempt to answer them. 12 Mr. Drager? 13 MR. DRAGER: I have a few. John Drager from 14 Telcordia again. 15 I heard that the test generator is not yet 16 chosen, I got that, or at least I inferred that. You 17 said you hadn't made it public. I guess that's my 18 question. Do you have a date when the test generator 19 will either be chosen or made public? Same thing to 20 me. 21 MR. LAKRITZ: They should be made public either at 22 the end of today or tomorrow. 23 MR. DRAGER: Thank you. 24 Another question, if I may. The Master Test 25 Plan that we are to respond to is alluded to be a draft, 26 and I'm aware that the industry, we're commenting on 27 it. Can you tell me if you now view the Master Test 28 Plan that we have as the final version or are there PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 8 1 changes that we would expect to see? 2 MR. LAKRITZ: It's not the final version. I can't 3 tell you whether there will or won't be changes until 4 the Commission has made up its mind on that or, more 5 precisely, the assigned commissioner. We're expecting 6 sometime within the next two weeks to release the final 7 Master Test Plan. We understand the closer that we 8 release that document to when the bids are due, the less 9 likely it is that vendors will be able to incorporate 10 that into their bid response, so we're not making it a 11 requirement. But obviously, if people are able to know 12 the differences, that's great, but it's not a 13 requirement. 14 MR. DRAGER: I have been doing a lot of reading -- 15 I've got yet another one. 16 MR. LAKRITZ: Sure. 17 MR. DRAGER: You sent out to us a score card by 18 which the respondents will be measured, and I read that 19 with considerable interest. But No. 10, which is the 20 one I thought was really interesting, was the price 21 equation. You say if the bid falls in within the 22 following range, you will assign a percentage based on 23 the following scale, but my copy had no scale. 24 MR. LAKRITZ: We're not making that scale public. 25 Because we feel -- 26 MR. DRAGER: Got it. 27 MR. LAKRITZ: -- it would slant the bids. 28 MR. DRAGER: So you've got it, but you're not sure PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 9 1 -- got it. Thank you. 2 MR. LAKRITZ: That way, in case we grossly 3 misestimated the price, either good or bad, it won't 4 affect people who bid. 5 Actually, I think Bonnie had a question. 6 MS. ALEXANDER: Another question. In that score 7 sheet that he was just mentioning, it does indicate one 8 of the questions or one of the scoring criteria is 9 knowledge of this industry and what have you, but also 10 independent from PacBell, No. 6.2. 11 MR. LAKRITZ: Okay. 12 MS. ALEXANDER: And also I don't recall where I was 13 reading it, but I was reading something that says, 14 asking, in the last two years have you ever had an 15 affiliation or work related to Pacific Bell, and it 16 gives the specifics and what have you. Is that a good 17 thing or is that a bad thing? Are you going to rate 18 that positively if someone has an affiliation that 19 understands Pacific Bell's internal operations, or 20 negatively? 21 MR. LAKRITZ: There's actually two questions 22 there. I'll handle 6.2, and then Ellen can handle the 23 other question regarding the listing of the contracts. 24 What we're looking for in 6.2 is we're 25 interested in making sure that the potential vendor 26 understands the current contractual relationship they 27 will be in with Pacific Bell, i.e., that Pacific Bell is 28 paying for the contract, but they're reporting solely PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 10 1 and exclusively to the California Public Utilities 2 Commission and to the assigned staff. 3 MS. ALEXANDER: Okay. 4 MR. LAKRITZ: Then with regards to the question 5 about contracts, Ellen can answer that. 6 MS. LEVINE: The question goes to whether there are 7 any conflicts of interest between the vendor and either 8 SBC or its affiliates, and/or the parties in the 271 9 proceeding. So we would -- I guess the closer the 10 relationship, for example, that a vendor might have with 11 SBC on the very matters for which we're seeking a 12 contract, the greater the conflict. At least that 13 situation would raise questions of whether there's a 14 conflict of interest. 15 MS. ALEXANDER: Whether there's a conflict of 16 interest, not if there is, because -- 17 MS. LEVINE: We certainly recognize that there will 18 be relationships between probably most of the vendors 19 and the parties in our 271 proceeding, so it's really 20 the nature of the relationships and how closely they're 21 related to the matters for which we're seeking 22 proposals. 23 MS. ALEXANDER: And I guess I'm trying to, to my 24 benefit, say the closer the relationship was to Pacific 25 Bell and the development of their local competition 26 products, the better it is for the consultant to know 27 where the holes in their system are, so to speak. I 28 don't know if I'm getting that across very well. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 11 1 MS. LEVINE: I guess we would want to know what 2 those relationships are and then we'll make an 3 assessment of whether -- 4 MS. ALEXANDER: How long ago it was. 5 MS. LEVINE: -- it creates a question of conflict 6 or bias really on the part of the vendor towards -- 7 MS. ALEXANDER: Right. 8 MS. LEVINE: -- the entity with which it 9 contracted. 10 MS. ALEXANDER: Right, okay. 11 MS. CLAVENNA: Leigh Ann Clavenna, Telecom 12 Technologies. 13 In relation to that, in Appendix B regarding 14 the 271 proceedings, is it a relationship with any of 15 those people only in California, or anywhere in the 16 U.S., or in the world? 17 MS. LEVINE: I believe it's the relationship with 18 those corporate entities. 19 MR. LAKRITZ: Right. It's primarily -- 20 MS. LEVINE: Primarily California is what we'd be 21 interested in. 22 MR. LAKRITZ: But obviously if you have overseas 23 contracts, that's a less critical nature and presents 24 far less problems for conflict, but I know that our -- 25 certain corporate entities will have national accounts 26 with, for instance, AT&T that would allow them to do 27 work in any jurisdiction, so then we may have questions 28 about the nature of the work that that vendor has done PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 12 1 in California. 2 MS. CLAVENNA: Okay. So would you want -- if we 3 hadn't done anything in California, but we'd done 4 something in Texas or Virginia, do you want us to 5 include that? 6 MS. LEVINE: Yes. 7 MR. LAKRITZ: Yes, particularly Texas, if it's SBC 8 or affiliates. 9 MS. CLAVENNA: What about relationships with any of 10 the other parties in here? For instance, MCI WorldCom, 11 AT&T, you'd like that also? 12 MS. LEVINE: The same. Particularly to the extent 13 that that work related to the type of work that we're 14 looking at here. 15 MS. CLAVENNA: Okay. 16 MR. SMITH: Jason Smith from Price Waterhouse 17 Coopers. 18 I have a couple of questions, again, on the 19 independence issue and relationships with other 20 parties. We have a number of audit relationships with 21 some of the parties. Do you perceive that as a conflict 22 of interest? 23 MS. LEVINE: I guess I would have to know more 24 about what you mean by the, "audit relationships." 25 Again, the closer the connection is to what we're 26 looking for here -- 27 MR. SMITH: Right. 28 MS. LEVINE: -- the more questions that PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 13 1 relationship will raise. 2 MR. SMITH: Okay. I actually mean the external 3 financial law that's required by SCC. 4 MS. LEVINE: Well, I would mention it. I don't see 5 that -- if it's totally unrelated to the matters here, I 6 don't believe that's going to create a disqualifying 7 conflict of interest. 8 MR. SMITH: Okay. The other question I had is in 9 the Appendix B there's several individuals who work 10 apparently for attorneys, law firms, but are not 11 necessarily associates of any particular companies, 12 per se, that we would expect to see in the proceedings. 13 Do we know who they are, or are they representing 14 companies that are already in the list or -- 15 MR. LAKRITZ: I think at this point that we're 16 mostly interested in the corporations that are listed in 17 Appendix B. 18 MR. SMITH: So we should ignore the law firms? 19 MR. LAKRITZ: Unless -- 20 MS. LEVINE: Unless you know that that law firm 21 represented a particular corporate entity. 22 MR. SMITH: Okay. 23 MR. DORRIAN: Michael Dorrian, Lockheed Martin. 24 I have a follow-up question to one of the 25 considerations raised before about the fact that the 26 Master Test Plan is still in draft form and that there 27 will be additional changes coming on between now and the 28 next two weeks. I take it the, all the bidders should PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 14 1 anticipate bidding on the draft MTP as it exists right 2 now, and anything between now and when the bids are due 3 should be excluded in that consideration. 4 For example, if you give us something on 5 Monday and it changes the scope of the work, should 6 we -- we should exclude that from our bidding process or 7 is there a cutoff time on that? 8 MR. LAKRITZ: I think at this point in time, and if 9 we change, in any way, this, we will let, clearly let 10 you know, but at this point in time you need bid only on 11 what's in the draft Master Test Plan. I don't know if 12 the Commission at this point envisions substantial 13 changes in terms of the scope of what the test 14 administrator needs to do. 15 MR. DORRIAN: I'll give you an example that sort of 16 pops out is the, the exit criteria for the test 17 execution phase, there's still deliberations as to 18 whether it should be validation or an audit, and that 19 would be coming out of the assigned commissioner. 20 MR. LAKRITZ: Right. There are a few key areas 21 that could substantially affect the work. I think, for 22 the purposes of this bidding, if you want to focus 23 solely on what's in the draft Master Test Plan, that's 24 certainly fine. If things have changed during the 25 intervening period, we may ask you during the interview 26 if you do have any thoughts about what this change would 27 do, generally, about what you're bidding on. 28 MR. DORRIAN: Okay. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 15 1 MR. SMITH: I have another question. Do you have a 2 general sense as to the overall timing of the test 3 effort itself during the actual execution phase? 4 MR. LAKRITZ: Well, we're expecting that most of 5 the effort will be conducted during the third and fourth 6 quarter of this year. So, as I mentioned earlier, we're 7 hopeful that the test will be able to start 8 approximately four to six weeks after we undertake a 9 test administrator. We're looking at about two months' 10 worth of testing two billing cycles as outlined in the 11 draft Master Test Plan. We expect that there will be a 12 period for the test administrator to prepare a report 13 afterwards, comments from parties, and then probably a 14 Commission decision. 15 MR. DORRIAN: Michael Dorrian, Lockheed Martin. 16 I have a recommendation for the staff to 17 consider. The reference point that the test 18 administrator/manager, RFP item 5 under the proposal 19 which reads, "Include the time and unit price." The 20 question is: Might it be appropriate to move the time 21 and unit price requirement to RFP item 6? In that way 22 the staff would have all the proposal price items in the 23 same proposal location to facilitate the evaluation 24 process. 25 MR. LAKRITZ: That's certainly fine. I mean, when 26 we laid out these criteria we did not have the 27 expectation that they were so rigid that you can only 28 answer the criteria in the number that they were put PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 16 1 forth here, but yes, if you want to put it together, it 2 makes good sense to put it all in the price proposal. 3 Make an active effort to read the complete vendor 4 proposal to make sure that they cover the information. 5 Okay. Mr. Drager, did you have a question? 6 MR. DRAGER: Yes. A couple more, if I may. John 7 Drager from Telcordia. 8 Okay. We also read that we should state 9 assumptions, etc., etc., and the assumptions in the 10 Master Test Plan as well as in the industry lead me to 11 question, lead me to the following question: How is the 12 Commission's attitude toward a time and materials quote 13 as opposed to a firm price quote? It seems to me that 14 you run the risk right now of agreeing on something 15 that's going to get done and then you're going to get up 16 scoped by a bunch of folks who gave you firm price 17 quotes, because in point of fact we weren't able to 18 foresee that. So the question is, what is your view 19 about a time and materials quote versus a firm price 20 quote? 21 MR. LAKRITZ: I think that at a minimum we want a 22 firm price quote, but if the vendor does choose to 23 provide us also with a time and materials quote or an 24 indication of if additional work is needed approximately 25 what the time and materials cost would be, that would 26 certainly be very helpful, but we still have the bottom 27 line requirement that we want to see a firm price quote 28 or a fixed price quote at this point in time. I mean, PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 17 1 certainly it would prove very useful and constructive 2 for us to get a sense of for additional or delays in the 3 project, what those costs would be, but at this point in 4 time -- 5 MR. DRAGER: Got another one. 6 The test administrator is assigned the 7 responsibility for designing test cases and scripts, 8 etc., etc., etc. The question is: Who approves them 9 and says, Okay, this is the ones that really are 10 necessary and sufficient? 11 The reason I ask, incidentally, has to do with 12 my reading of what goes on in other jurisdictions, both 13 New York and Texas being good examples, where late in 14 the game, for instance, various respondents have said, 15 You know, you guys just tested the wrong thing. So one 16 would think that certainly the test administrator is 17 going to come up with these test cases, I got that 18 part. But there ought to be some kind of buy in, if you 19 will. 20 MR. LAKRITZ: Sure. Well, I think that it's a 21 two-part answer. First is we're hoping that the test 22 administrators that are developing the test cases will 23 communicate some with interested CLECs to have a better 24 understanding of the scope of test cases under the 25 scenarios that will be established in the Master Test 26 Plan. And then I think, second, that a final set of 27 test cases will be approved, probably by the Commission 28 staff in consultation with the assigned commissioner. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 18 1 We understand that there is a need to establish a final 2 set of test cases before one enters the task, or it 3 certainly would be preferable. 4 MR. DRAGER: I've got another one. 5 There's another proceeding that you folks have 6 been enjoying on performance measures, and we read with 7 considerable interest the performance measurements 8 document. Now, that one says effectively that the 9 Commission has heard the competitive Lex and the 10 incumbent Lex talking about the 44 measures and have 11 said, You guys agree on all of these, it's now a done 12 deal, and you don't agree on these few and we're going 13 to give you some help. But also in the body of that 14 document it says, You guys, "you" being the competitive 15 Lex and the Pacific, are going to agree on local number 16 portability measures by July 1st, some billing stuff by 17 August 1st, etc. Now, all that stuff needs to be 18 cranked into the test planning. The question is, did 19 they do what they were supposed to do by July 1st, and 20 do you believe that they're going to reach agreement by 21 August 1st? 22 MR. LAKRITZ: I'm not aware if they reached their 23 July 1st deadline. Unfortunately, I haven't had an 24 opportunity to check up on that. I think that they will 25 likely reach agreement on L&P measures and my guess is, 26 they will probably also reach agreement on billing 27 measures, but I don't know what the current status of 28 their negotiations are. So I don't know, can't answer PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 19 1 your question definitively. 2 MR. DRAGER: Okay. 3 MR. DORRIAN: I'll take you off the hook for a 4 minute, catch your breath. 5 Related to an earlier question posed by the 6 Telcordia representative -- 7 Sorry, Michael Dorrian Lockheed Martin. 8 We have a reference in the -- again, on the 9 technical advisor, RFP item 5 proposal, which includes 10 the concept of time and unit price. Can you give a 11 definition of time and unit price and how it applies 12 under a firm expressed contract? Just trying to get a 13 common basis for understanding evaluation how the price 14 proposal evaluation may work. 15 MR. LAKRITZ: Our primary thought in that, 16 developing that particular criteria, was focusing in on 17 the price per unit of employee, senior consultant would 18 be X hundred dollars an hour, a junior consultant would 19 be X hundred dollars an hour. We're also interested, if 20 it's possible, and we realize that this may be too 21 difficult, but to estimate a weekly cost difference if 22 the test execution period extended a longer, several 23 more weeks, or if the period leading up to test 24 execution extends. We realize there's some difficulty 25 there because it's a little problematic to know who 26 you're going to have working at that point in time to 27 know how much additional costs per week would be, but if 28 vendors are able to provide that information, it PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 20 1 certainly is helpful and instructive for us, more so in 2 just getting a better understanding of what potential 3 additions to the test would cost. 4 MR. DORRIAN: A follow up to that -- 5 MR. LAKRITZ: Sure. 6 MR. DORRIAN: -- relating to the item 6.1.A of the 7 advisory criteria score sheet. There is a reference 8 point to a project budgeting process. Can you clarify 9 what the intention of that requirement is, is this for 10 project milestones with some kind of financial 11 milestones, and if it's a firm and fixed price? Just 12 for clarification how that may be evaluated, how the 13 staff would look to use that as a tool for project 14 management? 15 MR. LAKRITZ: I think, Michael, what we're 16 primarily interested in is making sure that whichever 17 vendor was of contract with has or will have in place at 18 the time of the contract will be executed a budget for 19 the project with time and financial milestones, so that 20 when this task is done we expect to spend approximately 21 this amount of money. So it's a tool for the staff to 22 be able to monitor the progress of the budget, both in 23 terms of deliverables, but also in terms of financials, 24 so it doesn't sneak up on us that, you know, there's 25 going to be a million dollar cost overrun is an 26 example. So we're interested in making sure -- we also 27 think it reflects back on the vendor's ability to be 28 able to control costs as the project goes on, and I'm PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 21 1 sure that anyone bidding on a potential, a very large 2 contract such as this one would put that type of process 3 in place anyhow just for their own internal use. 4 MR. DORRIAN: Absolutely. Thank you. 5 MR. SMITH: Jason Smith, Price Waterhouse Coopers. 6 I would like to know what the motivation was 7 behind combining the technical advisor and the test 8 administrator manager role as one firm doing both, both 9 functions. 10 MR. LAKRITZ: I think when we went into the 11 collaborative process to develop the draft Master Test 12 Plan, we envisioned that parties might have had a 13 preference to have three separate entities. As we got 14 into that process, it became clear that it was becoming 15 more problematic to separate the two roles than to just 16 combine the two roles. There was sort of a general 17 consensus in the room that not much was gained by having 18 three entities overseeing the process, that having the 19 test generator and another entity performing the rest of 20 the functions was sufficient. So we just decided to 21 combine the technical advisor and test administrator. 22 And we felt that there would be complimentary skills 23 between the test administrator and the technical 24 advisor, and we thought that the added confusion related 25 to having three entities talking together wasn't worth 26 the value. 27 MR. SMITH: If I can, just to follow up on that. 28 MR. LAKRITZ: Sure. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 22 1 MR. SMITH: We had originally thought that the 2 technical advisor would be in some other review role 3 over the test administrator manager, as well as in 4 particular with the test cases and so forth. Do you 5 perceive that as being any kind of conflict, even though 6 we just discussed the approval process by the 7 Commission, but do you perceive any conflict between 8 dividing the roles? 9 MR. LAKRITZ: No. 10 MR. DRAGER: Picking up on one of Jason's 11 questions. John Drager from Telcordia again. 12 Got the part of no need to separate the test 13 advisor and the test, technical advisor/test 14 administrator, but you still want separate responses? 15 Have you given thought to asking for single responses? 16 You're going to get a parallel response in a lot of 17 ways, you're asking a lot of the same questions. 18 MR. LAKRITZ: I think that may be a confusion we 19 unnecessarily created on our part. We were not 20 intending for it to be separate responses. 21 MR. DRAGER: Let me restate that then. You are 22 looking for a single response for both roles in a single 23 document? 24 MR. LAKRITZ: Yes, we're looking for a single 25 response for both roles in a single document. 26 MR. DRAGER: Thank you. 27 MR. LAKRITZ: And we're sorry for any confusion 28 that briefly caused anyone. That certainly was not our PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 23 1 intention. 2 MS. CLAVENNA: Leigh Ann Clavenna, Telecom 3 Technologies. 4 Are you looking for separate pricing or are 5 you looking for one price for both of the roles? 6 MR. LAKRITZ: I think we're still interested in 7 seeing separate pricing, but it seems unnecessary to do 8 a -- to duplicate the qualifications and all of that 9 seems unnecessary. 10 MS. CLAVENNA: So you want the pricing broken down 11 as two separate entities, but just one document, one 12 answer? 13 MR. LAKRITZ: You can do one project plan and that 14 took care of roles for both. 15 MS. CLAVENNA: Another question. Throughout the 16 Master Test Plan, you had indicated people needed to be 17 on location. Are there multiple locations, similar 18 locations, and would they be required to be on location 19 throughout the entire test effort? 20 MR. LAKRITZ: It depends upon the location. For 21 some locations they will need to be there for the period 22 of the test execution, for other types of efforts they 23 will not need to be there for the entire time. There 24 probably will be, no doubt there will be several 25 locations. I don't know all of them yet. Most of the 26 test generators have proposed establishing testing 27 facilities in a separate location than Pacific Bell's 28 location. Some of those proposals include facilities in PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 24 1 state, some of them include facilities out of the 2 state. And since I can't tell you who the winning 3 bidder is yet, I can't tell you where that location is. 4 So that would certainly be more -- there probably will 5 be some observation, I'm sure, at some of Pacific's 6 facilities, both probably at central offices and where 7 they process their orders. 8 MS. CLAVENNA: Basically, the question is: Do you 9 want to include in the pricing for travel and other 10 expenditures related to that? And we just wanted to try 11 to get an idea as to how many places people need to be 12 simultaneously throughout the test effort. 13 MR. LAKRITZ: Unfortunately, we don't have a good 14 handle on that. I realize that presents a problem for 15 you. If you want to make assumptions about a price for 16 one location, you can multiply from multiple locations, 17 that's certainly fine. I just think at this point we're 18 not -- part of it would be what the test administrator 19 thinks they need to observe too, so that's why I'm a 20 little -- kind of hedging around the edges. They may 21 not -- one potential vendor may not think they need to 22 observe each and every location, and others may think 23 they need to. 24 MS. CLAVENNA: One more question. Regarding the 25 resumes for item No. 4, for the engagement team. Are 26 you looking for resumes for every single person that 27 would be involved in that or just the key players? 28 MR. LAKRITZ: I think we're primarily interested in PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 25 1 the key players, and then if you can provide 2 identification of the types of employees that would be 3 working on the rest of the project, not necessarily 4 their resumes but sort of general jobs. 5 MS. CLAVENNA: Thank you. 6 MR. LAKRITZ: Sure. 7 Mr. Drager? 8 MR. DRAGER: John Drager from Telcordia one more 9 time. 10 Who will see the responses that we provide to 11 you? We will give you a response to your RFP, who will 12 be reading them? This is a question of organizations. 13 Much of what you do becomes public record, and that's 14 where I'm coming from in that question. 15 MR. LAKRITZ: The four of us will be the primary 16 reviewers, and then, as with the test generator, when we 17 announce who the winning bidder is, those documents will 18 be available through local records request, so. 19 MR. DRAGER: Will we have the opportunity, if we 20 are selected, to edit some of that down? If there were 21 proprietary information inside it that we don't chose to 22 share with the rest of the industry, will we have an 23 opportunity to edit it down? 24 MR. LAKRITZ: Yes, but I should forewarn you that 25 the proprietary information cannot include price. 26 MR. DRAGER: Got it. 27 MR. LAKRITZ: So if there are -- it's a little 28 easier with the test generator, it's easier to PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 26 1 understand. They're describing software they're going 2 to be using to us and describing how that software 3 works. Our intention is to make as much of the 4 proposals available publicly as we can. And I should 5 clarify that both winning and losing proposals will be 6 made public, as well as the score sheets, so the entire 7 package is available for people who have nothing better 8 to do with their time. 9 MS. CLAVENNA: Leigh Ann Clavenna, Telecom 10 Technologies. 11 You keep mentioning the score card. Is that 12 available on your Web site? 13 MR. LAKRITZ: Yes, it is. 14 MS. CLAVENNA: Okay. Thank you. 15 MR. SMITH: Jason Smith from Price Waterhouse 16 Coopers. 17 Just curious as to the original time line was 18 extended for the actual, from the original RFP for the 19 test administrator manager and this bidders conference 20 was added and the roles were combined. Were there any 21 particular reasons behind extending the time line, 22 besides perhaps changing the roles, the nature of the 23 roles and so forth? Are there any other issues to be 24 aware of? 25 MR. LAKRITZ: We're mostly primarily interested in 26 making sure that the potential bidders have as much 27 information as they would need to be able to provide a 28 clean and clear bid to us. Because we realize it's a PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 27 1 very complex project and it's probably a 150-page draft 2 Master Test Plan, and that there are -- certainly there 3 are areas that can be clearer than other areas. 4 MR. DORRIAN: Michael Dorrian, Lockheed Martin. 5 Just looking for more of a clarification about 6 we're now combining the technical advisor with the test 7 administrator manager. What kind of role does the 8 Commission intend for the technical advisor to play? Is 9 it going to be a very active participation? Is it going 10 to be a subject matter available to the Commission to 11 comment about difficulties or answer questions the staff 12 may have? Can you provide background and clarification 13 about what you intend that role to be? 14 MR. LAKRITZ: Sure. If you look on Page 3 of the 15 RFP, that's the July 6 addendum, it lists approximately 16 10 activities that we were expecting the technical 17 advisor to be participating in. Some of them are more 18 of the review and audit type and some of them are more 19 direct hands-on. Certainly one of them is to provide 20 technical advice to the Commission staff, and that would 21 probably be more or less on an on-call basis. Other of 22 the activities, such as evaluating the documentation and 23 reviewing the test cases and providing some oversight of 24 the test generator are -- and analyzing the data report, 25 are more hands-on activities that will probably have 26 specific schedules and time lines involved in them, so 27 it's -- I think for our purposes, at this point in time 28 we're not really viewing them as being two separate PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 28 1 roles, but for the purposes of the fact that that's the 2 way the draft document was written, we're sort of 3 keeping that in name on paper, but we understand that an 4 entity that's going to be undertaking this process will 5 be looking at the entire range of responsibilities that 6 they have. 7 MR. DORRIAN: Would the people serving as the 8 technical advisor be separate and distinct from the test 9 administrator? Are you looking to have a more or less a 10 swat team that you would deploy? 11 MR. LAKRITZ: I don't think we're looking for a 12 separate engagement team, so it could be the same set of 13 employees or, you know, in some cases where one is 14 providing on-site observing of the actual test, the 15 potential vendor may decide that two sets of employees 16 is better for that activity, they may decide that one is 17 sufficient, and they have internal controls. I think 18 that it's changed a little bit and unfortunately we 19 probably didn't write the RFP as clear as we could 20 have. All that is sort of revolving for us, to be 21 honest. 22 MR. AMATO: The listing of tasks are also found in 23 the Master Test Plan on Section 5.2.3. 24 MR. McLELLAN: John McLellan with Hewlett Packard. 25 I have to admit I still struggle over that 26 section and when I look at it, please take no offense, 27 words like "conflict of interest" and "redundancy" come 28 to mind. I'm still struggling with that. Maybe a PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 29 1 little bit more discussion to help us feel a little bit 2 better about that. I didn't -- we were certainly 3 thinking about the idea of two different people, 4 wondering if there were any Chinese walls that needed to 5 be put up, because you're talking about almost an 6 auditing function, okay? 7 In other words, the technical advisor is 8 really reviewing the work of the test administrator 9 manager and there is potential conflict of interest and, 10 as I say, even potential redundancy. Now, especially if 11 they are even the same person. So I'm even more 12 confused. 13 MR. LAKRITZ: If they're the same person, at least 14 it removes redundancy. 15 MR. McLELLAN: Yes, that's true. 16 (Laughter) 17 MR. LAKRITZ: I hope. I wouldn't want to get two 18 bills for the same person. Probably it's best for us to 19 take that question back, and we can provide an 20 additional clarification on Monday when we do the rest 21 of it. Because I can certainly understand where your 22 confusion is coming from, and it was a case of a moving 23 target, I think. 24 MR. McLELLAN: Sure. Thank you. 25 MR. DRAGER: I do have one more question. This is 26 based on what Mike said a little while ago and your 27 response to it that has to do with the draft Master Test 28 Plan and whatever you're sending out on Monday. The PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 30 1 question is: Do I include that as part of my response 2 or not, what you're sending out on Monday? You alluded 3 to it already, you told me what to expect, project plan 4 roles for 5.2.4 and clarification on redundancy, etc., 5 so I understand, I think, what I'm going to get, but I 6 don't have it yet. And the question is: Should my 7 response take that into account? 8 MR. LAKRITZ: Yes, because what's being sent out on 9 Monday is actually a request for the vendors to provide 10 a project plan and to describe how they're going to do 11 the roles, roles and/or tasks described in Section 12 5.2.4. So yes, we would hope that that would be 13 included in the vendor's proposal. 14 I couldn't imagine we're done yet. 15 MR. SMITH: Jason Smith, Price Waterhouse Coopers 16 again. 17 Just to clarify again. You did say the test 18 generator will be announced either tomorrow or Monday, 19 is what you said? 20 MR. LAKRITZ: Either today or tomorrow. 21 MR. SMITH: Today or tomorrow, okay. And when you 22 announce that, are you going to give us an inclination 23 about the facilities and the sites that we talked about 24 before? 25 MR. LAKRITZ: What we can do is we can make 26 available after you submit a public records request 27 their proposal which would include at least their 28 facility site where they plan on setting up, you know, PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 31 1 in essence their software and their team of individuals 2 to input things, but some of the other sites are unclear 3 to us right now and are also, as I said, dependent upon 4 what the test administrator, what types of site 5 observation they think they need to do. For instance, 6 if a vendor decides they need to be in every central 7 office whenever an order potentially goes through, it's 8 a little hard. 9 MR. DORRIAN: Michael Dorrian, Lockheed Martin. 10 I'm sorry, I don't have the actual cite, but 11 the -- I'm assuming it's in the pricing proposal 12 section, but there are words to the effect that vendors 13 should detail any of the assumptions going into the 14 price bid. Can you provide some clarity or some 15 examples as to what those assumptions would look like? 16 Would they be unit costing and time definitions? 17 MR. LAKRITZ: I can give you an idea from what we 18 experienced in the test generator proposals. Some of 19 them would put in a set of assumptions that would 20 clarify that they do not have to develop the test cases, 21 that Pacific would pay for any interconnecting 22 facilities that were necessary to get to wherever they 23 were going to establish their test generation device, 24 those types of things. So basically assumptions about 25 what types of information would be provided to them and 26 what types of facilities. 27 MR. DORRIAN: Okay. 28 MR. LAKRITZ: So it's more along the lines of that PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 32 1 than it was along the lines of unit pricing. 2 MR. DORRIAN: Okay. Does that also include the 3 quality of the information provided by the incumbent? 4 MR. LAKRITZ: Sure, yes. Some certainly made 5 references to that. 6 MR. DORRIAN: One more follow up on that. There is 7 a reference in MTP, Master Test Plan, Section 5.2.2 that 8 states, "The role of Pacific is to: Provide the Pacific 9 OSS environment to be used for the test." My question 10 is: How much space and facilities would Pacific provide 11 for the test administrator to use during the test 12 development phase or the test execution phase? 13 MR. LAKRITZ: What type of facilities are you 14 referring to, Mr. Dorrian? 15 MR. DORRIAN: Facilities that need to be 16 collocated, will they be providing working space, will 17 they be providing telephone facilities? It seems to be 18 fairly wide open that -- almost carte blanche, but we 19 define to the incumbents what we need at designated 20 locations and I would have to imagine there may be some 21 restrictions to the ability of Pacific to provide this, 22 based on the location. 23 MR. LAKRITZ: Sure. The best thing to do is to 24 clarify in your response what assumptions you're making 25 about how much space the facility would provide to you. 26 We certainly haven't reached that level of detail in our 27 discussions. 28 MR. DORRIAN: Sure. PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 33 1 MR. SMITH: Jason Smith, Price Waterhouse Coopers. 2 Will the firm selected to administer the test 3 be excluded from doing any of the work with Pacific Bell 4 or SBC during the test period or any time thereafter? 5 And if so, what kind of work would be excluded? 6 MR. LAKRITZ: Just one second on this one. We have 7 sort of struggled with this question. 8 MR. SMITH: Sure. 9 MR. LAKRITZ: I can tell you our current answer, 10 which I can't guarantee won't change, but currently what 11 we would envision is that as part of the contract we 12 would require the vendor to not engage in any activities 13 related to Pacific Bell's 271 compliance filing, or any 14 related activity either at this Commission or any other 15 state commission that the SBC is part of or at the FCC. 16 Similarly, there'd be the same sort of 17 requirement upon CLECs or other competitive carriers. 18 And that requirement would -- primarily will ask for the 19 period of until Pacific receives its 271 approval. 20 Which is a little open ended, so we could discuss 21 certain time frames that are involved. But at this 22 point in time, we're hopeful that that will be a 23 relatively short period, so it will not present a major 24 barrier. 25 We also have concerns, although we feel that 26 those could be addressed through contractual terms, 27 about the same, the vendor providing similar services in 28 terms of OSS development with individual CLECs and with PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 34 1 Pacific Bell, but those I think we can handle. They're 2 not prohibited, I just think there's concerns we have 3 about the nature of the type of work that would be 4 done. We certainly don't have any intention of, for 5 instance, preventing someone from doing work in an 6 unrelated field like financial audits or helping them 7 with numbering or helping them with, you know, 8 performing customer billing for their own customers. So 9 those types of things are not such a large concern for 10 us. Primarily, we just want to prevent, you know, 11 seeing the same consultant's name on that testation file 12 by either Pacific or by some competitive carrier and on 13 our report. This is just to remove sort of obvious 14 conflicts that could potentially come up. 15 MR. McLELLAN: John McLellan with Hewlett Packard. 16 I realize this may be a bit out of scope but 17 yet related. Has the Commission given any thought to 18 ongoing verification testing once the 271 approval takes 19 place? 20 MR. LAKRITZ: We haven't thought about that. 21 MR. McLELLAN: Okay. 22 MR. LAKRTIZ: Bonnie? 23 MS. ALEXANDER: Bonnie Alexander, Network 24 Intelligence. 25 Under the Master Test Plan and the development 26 of test scenarios, you're setting yourself into a closed 27 environment, in my opinion, using test data, test 28 situations. I'm wondering if it might be a thought for PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 35 1 you to add into this test plan actual live data and 2 actual order data, because I have been affiliated with 3 Lex and Verigate, and it's been about a year and a half 4 ago, but it was on their test, in their test database, 5 and those products were great. But it doesn't work in 6 the real world. And you're setting yourself up for it 7 with this Master Test Plan to test load and port and all 8 these things. They work great in a test. They have 9 test databases that zip it right through. 10 MR. LAKRITZ: Well, I should clarify that the test 11 orders will be sent over during the same time period the 12 live production orders will be sent. There's going to 13 be four operator codes, so it will make it difficult for 14 Pacific to ascertain whether the order coming over is a 15 test order or a live order, and so it's being tested in 16 a production environment. I mean -- 17 MS. ALEXANDER: Okay. But in fact, a test order 18 will go through and are you actually going to install a 19 phone in someone's house as a test order? 20 MR. LAKRITZ: Yes. 21 MS. ALEXANDER: You do intend to follow through to 22 the means of actually doing a loop and port in someone's 23 house? 24 MR. LAKRITZ: For the functionality test, we will 25 be assigning locations for actual end-to-end 26 provisioning, including the generation of bills. 27 MS. ALEXANDER: And the Pacific Bell and the 28 WorldCom, or whomever else, CLEC, Lex will not know if PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 36 1 it's a test order or a live order from CLEC? Because 2 you have to make it look like it comes from a CLEC to go 3 through the Pacific Bell system -- 4 MR. LAKRITZ: Right. 5 MS. ALEXANDER: -- and get provision. 6 MR. LAKRITZ: I can't honestly say at this point 7 they won't be able to tell. Certainly on capacity test 8 day when the orders peak in the first hour, they're 9 probably going to know they're in a test environment. I 10 mean, there's a certain limit as to the blindness that 11 you can create. I think certainly the Texas Commission 12 will learn that, at some point in time you struggle and 13 if you're going to be, you know, increasing their system 14 load by 10 or 20 times the current level, they're 15 probably going to know something is up. 16 MS. ALEXANDER: That also points to the future, 17 when the future growth does occur because of it working. 18 MR. LAKRITZ: Certainly we're trying to make it as 19 reasonably -- 20 MS. ALEXANDER: Real. 21 MR. LAKRITZ: -- objective as we can when we that 22 see there's a balance there. And it's pretty hard. 23 Actually, I prefer Pacific didn't comment on this 24 issue. 25 MR. HOUSTON: That's fine. 26 MS. CLAVENNA: Leigh Ann Clavenna, Telecom 27 Technologies. 28 Who is going to be responsible for actually PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 37 1 procuring these end users and having people volunteer 2 for the installs at their locations? 3 MR. LAKRITZ: It's going to be -- the end users 4 will be a combination of Pacific Bell end user sites and 5 those established by the test administrator. So the 6 test administrator will have final responsibility. 7 MS. CLAVENNA: Okay. 8 MR. SMITH: Jason Smith, Price Waterhouse Coopers 9 again. 10 To what extent of funding will the parties in 11 the proceeding have any input into the selection of the 12 test administrator, technical advisor? 13 MR. LAKRITZ: They will have no input into it. 14 MR. AMATO: Is that it? 15 MR. LAKRITZ: Thank you very much. We look forward 16 to seeing your bids on August 10th. We will be 17 providing a transcript, an E-mail copy of the transcript 18 to everyone who did the sign up list. I encourage you 19 to sign up. If there is anyone you know that was not 20 here, you can either feel free to forward the transcript 21 on to them or you can have them contact us, either way. 22 And we will provide a further clarification early next 23 week about the test administrator and the technical 24 advisor role. 25 Are there any additional questions? Thank you 26 very much. 27 (Whereupon, at the hour of 2:30 p.m., the 28 Bidders Conference was concluded.) PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA 38 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 PUBLIC UTILITIES COMMISSION, STATE OF CALIFORNIA SAN FRANCISCO, CALIFORNIA