标签归档：protobuf

程序猿

bang dream proto相关

2018年4月12日 esterTion 2条评论

本文相关轮子： proto读取导出 | 定时任务本体 | 数据库对比主数据

A python + libil2cpp.so(Android) dumper:

Article: Generating Proto File For Banddream | DNARoma

gist

因为上周的一个想法，写出了redive和cgss的master数据库diff库，还造了一波bundle解压的轮子

redive第三个号被制裁了，平稳科技流也开始查了

土豆的内容星月大佬写离线的时候已经表示过master不可行，都是登录时返回部分master，所以就可以放弃了

然后就是bang了，开搞前我还又去debug了一下，结果看到密钥瞬间才想起power之前给我说过

炸梦通信是protobuf，包括master和manifest也是，所以就必须要提取原始关系才行了。我刚开始还想尝试暴力提取数据，但是内结构和字符串都是wireType 2，没法区分

在之前我已经在Prefare博客上看到过说dump属性什么的，dnspy改了之后也没搞懂怎么看。后来在评论区看到回复才找到

Perfare：邦邦用的是protobuf-net不是Google.Protobuf，不需要.proto文件，你只需要找到带ProtoContractAttrbuit的类并还原出properties上ProtoMemberAttribute的值就可以序列化了

知道入口后就开始观察了，搜索了一波然后看到master数据的类是SuiteMasterGetResponse（基本上各种proto类都叫GetResponse）

观察了下结构之后就开始写proto dumper了，以SuiteMasterGetResponse为入口，循环匹配含有 [ProtoMemberAttribute] 的成员，然后匹配子类递归。匹配中就需要特殊处理，比如xxxx[]就是一个数组，List<xxxx>也是一个数组，Dictionary2<xxx, yyy>就是键值对，观察了下数据分布这里的结构是

message entries {
  type1 key = 1;
  type2 value = 2;
}

message entries {

type1 key = 1;

type2 value = 2;

}

即key+value形式

名称和类型之类的导出做完之后，就是tag id了。tag id并不是连续数，所以就需要读取原始数据。每个成员的ProtoMember属性都带一个地址，这个地址是一个函数，每个函数看起来都是这样的：

__text:0000000101320B5C sub_101320B5C                           ; DATA XREF: __const:000000010256B948↓o
__text:0000000101320B5C LDR             X8, [X0,#8]
__text:0000000101320B60 LDR             X0, [X8]
__text:0000000101320B64 MOV             W1, #1
__text:0000000101320B68 MOV             X2, #0
__text:0000000101320B6C B               ProtoMemberAttribute$$.ctor
__text:0000000101320B6C ; End of function sub_101320B5C

__text:0000000101320B5C sub_101320B5C ; DATA XREF: __const:000000010256B948↓o

__text:0000000101320B5C LDR X8, [X0,#8]

__text:0000000101320B60 LDR X0, [X8]

__text:0000000101320B64 MOV W1, #1

__text:0000000101320B68 MOV X2, #0

__text:0000000101320B6C B ProtoMemberAttribute$$.ctor

__text:0000000101320B6C ; End of function sub_101320B5C

可以明显看出是个只有5条指令的函数，而id明显就是第3条，那么就只需要读取出立即数就行了

结果这个可以说是整个读取里面最大的坑了。因为某种优化，编译器输出的立即数赋值指令有两种：

movk Rd HALF
xx1x 0010 1xxi iiii iiii iiii iiid dddd

orr Rd_SP Rn LIMM
x01x 0010 0Nii iiii iiii iinn nnnd dddd

movk Rd HALF

xx1x 0010 1xxi iiii iiii iiii iiid dddd

orr Rd_SP Rn LIMM

x01x 0010 0Nii iiii iiii iinn nnnd dddd

movk指令的话，很简单，匹配中间的i位就可以了

但是orr整整卡了我两个小时，在群里大佬stat带领下才逐渐搞明白读取方法

简单讲，从第二字节开始，第一位为0，第二位N表示是否64位，3-8位为immr——旋转数，第三字节1-6为imms——大小数。此处nnnnn恒为11111，表示W31寄存器，即WZR，或读0。立即数结尾imms位设置为1，然后循环右移immr位，即为所读立即数

function readInst($inst) {
  $chk = $inst & 0x6f800000;
  if ($chk == 0x42800000) {
    $imme = ($inst & 0x1fffe0) >> 5;
    $dest = $inst & 0x1f;
    //echo "\tmov \tW${dest}, #${imme}\n";
    return $imme;
  } else if ($chk == 0x22000000) {
    $N    =($inst & 0x400000) != 0;
    $immr =($inst & 0x3f0000) >>16;
    $imms =($inst &   0xfc00) >>10;
    $dest = $inst &     0x1f;

    if ($N) {
      $size = 64;
    } else {
      $size = 32;
      $mask = 0x20;
      while($size > 1) {
        if (($imms & $mask) == 0) break;
        $size /= 2;
        $mask /= 2;
      }
    }

    $S = $imms+1;
    $pattern = bindec( str_repeat(str_pad(str_repeat('1', $S), $size, '0', STR_PAD_LEFT), 32 / $size));
    $imme = rotate($pattern, $immr);
    //echo "\torr \tW${dest}, W31, #${imme}\n";
    return $imme;
  } else {
    echo decbin($inst)."\n";
    throw new Exception('unknown');
  }
}

function readInst($inst) {

$chk = $inst & 0x6f800000;

if ($chk == 0x42800000) {

$imme = ($inst & 0x1fffe0) >> 5;

$dest = $inst & 0x1f;

//echo "\tmov \tW${dest}, #${imme}\n";

return $imme;

} else if ($chk == 0x22000000) {

$N =($inst & 0x400000) != 0;

$immr =($inst & 0x3f0000) >>16;

$imms =($inst & 0xfc00) >>10;

$dest = $inst & 0x1f;

if ($N) {

$size = 64;

} else {

$size = 32;

$mask = 0x20;

while($size > 1) {

if (($imms & $mask) == 0) break;

$size /= 2;

$mask /= 2;

}

$S = $imms+1;

$pattern = bindec( str_repeat(str_pad(str_repeat('1', $S), $size, '0', STR_PAD_LEFT), 32 / $size));

$imme = rotate($pattern, $immr);

//echo "\torr \tW${dest}, W31, #${imme}\n";

return $imme;

} else {

echo decbin($inst)."\n";

throw new Exception('unknown');

}

这里本来imms还表示pattern大小，但其实也没做出来，因为都是32位的pattern。

搞定tag id读取之后就是一路爽了，稍作修改就能输出规范proto。导出代码gist

造完这么个奇怪的轮子之后又得继续造读取轮子。现有的开源实现没找到能快捷转换protobuf到php数组的，估计也是php类型太少会导致格式信息丢失吧。拿出之前读b站弹幕的轮子改了改，换了最近写的FileStream轮子读取内容（~~Utaha：你要记住自己的本质是一个轮子工（~~），读完之后再进行字典键值对处理，最后又一个自定义json输出美化的轮子

高潮来了

写完逻辑代码后跑了一遍，但是我在第二遍才加了core.filemode=false，一堆mode change，于是…

一秒爽到

当场恢复了message导出代码，但是主逻辑不见了。今天早上花了一上午grep /dev/sdb1恢复（msys2竟然有这个设备映射），但是p都没找着。于是，又重新造了一遍自己刚造完的轮子（~~Utaha：你要记住自己的本质是一个轮子工（~~）

这次记录的基本上就这些，还是要感谢各位前辈造好的轮子，我才能这么爽到（x

程序猿

关于b站新弹幕格式.pb

2017年2月19日 esterTion 2条评论

格式为google protocol buffer (protobuf)

地址格式为{cid}-{part}.pb，每part包含3分钟的弹幕内容，即边播放边加载模式，如-3.pb包含第6-9分弹幕

在bilibiliPlayer.min.js中有大部分field id的定义，但文件中出现了id12-14

pb格式弹幕 Field常量
1  row (Row)
2  chat_server (string)
3  chat_id (varint32)
4  mission (varint32)
5  max_limit (varint32)
6  source (string)
7  ds (varint32)
8  de (varint32)
9  max_count (varint32)
10 realname (varint32)
11 sectionlen (varint32)(一般为180)
//猜测id
12 duration (float)
13 total_count (varint32)
14 pb_version (varint32)(目前常量1)

pb格式弹幕row Field常量
1  playtime (float)
2  mode (varint32)
3  fontsize (varint32)
4  color (varint32)
5  times (varint32)
6  poolid (varint32)
7  hash (string)
8  dmid (varint32)
9  msg (string)
10 uid (varint32)(暂未出现)
11 uname (string)(暂未出现)

pb格式弹幕 Field常量

1 row (Row)

2 chat_server (string)

3 chat_id (varint32)

4 mission (varint32)

5 max_limit (varint32)

6 source (string)

7 ds (varint32)

8 de (varint32)

9 max_count (varint32)

10 realname (varint32)

11 sectionlen (varint32)(一般为180)

//猜测id

12 duration (float)

13 total_count (varint32)

14 pb_version (varint32)(目前常量1)

pb格式弹幕row Field常量

1 playtime (float)

2 mode (varint32)

3 fontsize (varint32)

4 color (varint32)

5 times (varint32)

6 poolid (varint32)

7 hash (string)

8 dmid (varint32)

9 msg (string)

10 uid (varint32)(暂未出现)

11 uname (string)(暂未出现)

其中varint32为变长度int32格式，具体为每字节只有后7位存储数据，第1位为指示位，为1时继续读取下一字节

function readVarInt32($fp){
	$b=0;
	$result=0;
	
	do{
		$b=unpack('C',fread($fp,1))[1];
		$result  = $b & 0x7f;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<7;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<14;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<21;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<28;
		if(!( $b & 0x80 ))
			break;
		
		for($i=0;$i<5;$i++){
			$b=unpack('C',fread($fp,1))[1];
			if(!( $b & 0x80 ))
				break;
		}
	}while(false);
	return $result;
}

function readVarInt32($fp){

$b=0;

$result=0;

do{

$b=unpack('C',fread($fp,1))[1];

$result = $b & 0x7f;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<7;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<14;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<21;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<28;

if(!( $b & 0x80 ))

break;

for($i=0;$i<5;$i++){

$b=unpack('C',fread($fp,1))[1];

if(!( $b & 0x80 ))

break;

}

}while(false);

return $result;

}

文件读取代码样本

<?php
/*
 BiliBili pb danmaku reader
 @author esterTion(esterTionCN@gmail.com)
*/
$pb=fopen('http://comment.bilibili.com/14328539-1.pb','r');

function readVarInt32($fp){
	$b=0;
	$result=0;
	
	do{
		$b=unpack('C',fread($fp,1))[1];
		$result  = $b & 0x7f;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<7;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<14;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<21;
		if(!( $b & 0x80 ))
			break;
		
		$b=unpack('C',fread($fp,1))[1];
		$result |= ($b & 0x7f)<<28;
		if(!( $b & 0x80 ))
			break;
		
		for($i=0;$i<5;$i++){
			$b=unpack('C',fread($fp,1))[1];
			if(!( $b & 0x80 ))
				break;
		}
	}while(false);
	return $result;
}
function readString($fp){
	$strLength=readVarInt32($fp);
	return fread($fp,$strLength);
}
$types=array(
	'VARINT'=>0,
	'FIXED64'=>1,
	'DELIMITED'=>2,
	'START_GROUP'=>3,
	'END_GROUP'=>4,
	'FIXED32'=>5
);
$rowCount=0;
while(true){
	$id=fread($pb,1);
	if($id==='')
		break;
	$id=unpack('C',$id)[1];
	$type = $id & 7;
	$id = $id >> 3;
	switch($id){
		case 1:
			$rowLength=readVarInt32($pb);
			echo 'row length:'.$rowLength.' ';
			$end=ftell($pb)+$rowLength;
			$row=array();
			while(ftell($pb) < $end){
				$id=unpack('C',fread($pb,1))[1];
				$id = $id >> 3;
				switch($id){
					case 1:
						$row['playtime']=unpack('f',fread($pb,4))[1];
					break;
					case 2:
						$row['mode']=readVarInt32($pb);
					break;
					case 3:
						$row['fontsize']=readVarInt32($pb);
					break;
					case 4:
						$row['color']=substr(pack('N', readVarInt32($pb) ),2);
					break;
					case 5:
						$row['timestamp']=readVarInt32($pb);
					break;
					case 6:
						$row['poolid']=readVarInt32($pb);
					break;
					case 7:
						$row['hash']=readString($pb);
					break;
					case 8:
						$row['dmid']=readVarInt32($pb);
					break;
					case 9:
						$row['msg']=readString($pb);
					break;
					case 10:
						$row['uid']=readVarInt32($pb);
					break;
					case 11:
						$row['uname']=readString($pb);
					break;
				}
			}
			print_r($row);
			$rowCount++;
		break;
		case 2:
			echo '[chat_server] => '.readString($pb)."\n";
		break;
		case 3:
			echo '[chat_id] => '.readVarInt32($pb)."\n";
		break;
		case 4:
			echo '[mission] => '.readVarInt32($pb)."\n";
		break;
		case 5:
			echo '[max_limit] => '.readVarInt32($pb)."\n";
		break;
		case 6:
			echo '[source] => '.readString($pb)."\n";
		break;
		case 7:
			echo '[ds] => '.readVarInt32($pb)."\n";
		break;
		case 8:
			echo '[de] => '.readVarInt32($pb)."\n";
		break;
		case 9:
			echo '[max_count] => '.readVarInt32($pb)."\n";
		break;
		case 10:
			echo '[realname] => '.readVarInt32($pb)."\n";
		break;
		case 11:
			echo '[sectionlen] => '.readVarInt32($pb)."\n";
		break;
		default:
			switch($type){
				case 0:
					echo '--Unknown ID '.$id.' with varint32 value '.readVarInt32($pb)."\n";
				break;
				case 1:
					echo '--Unknown ID '.$id.' with fixed 64bit data hex '.bin2hex(fread($pb,8))."\n";
				break;
				case 5:
					echo '--Unknown ID '.$id.' with fixed 32bit data hex '.bin2hex(fread($pb,4))."\n";
				break;
				default:
					echo '--Unknown ID '.$id.' with type '.$type."\n";
			}
	}
}
echo $rowCount.' rows in file'."\n";

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

<?php

BiliBili pb danmaku reader

@author esterTion(esterTionCN@gmail.com)

$pb=fopen('http://comment.bilibili.com/14328539-1.pb','r');

function readVarInt32($fp){

$b=0;

$result=0;

do{

$b=unpack('C',fread($fp,1))[1];

$result = $b & 0x7f;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<7;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<14;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<21;

if(!( $b & 0x80 ))

break;

$b=unpack('C',fread($fp,1))[1];

$result |= ($b & 0x7f)<<28;

if(!( $b & 0x80 ))

break;

for($i=0;$i<5;$i++){

$b=unpack('C',fread($fp,1))[1];

if(!( $b & 0x80 ))

break;

}

}while(false);

return $result;

}

function readString($fp){

$strLength=readVarInt32($fp);

return fread($fp,$strLength);

}

$types=array(

'VARINT'=>0,

'FIXED64'=>1,

'DELIMITED'=>2,

'START_GROUP'=>3,

'END_GROUP'=>4,

'FIXED32'=>5

);

$rowCount=0;

while(true){

$id=fread($pb,1);

if($id==='')

break;

$id=unpack('C',$id)[1];

$type = $id & 7;

$id = $id >> 3;

switch($id){

case 1:

$rowLength=readVarInt32($pb);

echo 'row length:'.$rowLength.' ';

$end=ftell($pb)+$rowLength;

$row=array();

while(ftell($pb) < $end){

$id=unpack('C',fread($pb,1))[1];

$id = $id >> 3;

switch($id){

case 1:

$row['playtime']=unpack('f',fread($pb,4))[1];

break;

case 2:

$row['mode']=readVarInt32($pb);

break;

case 3:

$row['fontsize']=readVarInt32($pb);

break;

case 4:

$row['color']=substr(pack('N', readVarInt32($pb) ),2);

break;

case 5:

$row['timestamp']=readVarInt32($pb);

break;

case 6:

$row['poolid']=readVarInt32($pb);

break;

case 7:

$row['hash']=readString($pb);

break;

case 8:

$row['dmid']=readVarInt32($pb);

break;

case 9:

$row['msg']=readString($pb);

break;

case 10:

$row['uid']=readVarInt32($pb);

break;

case 11:

$row['uname']=readString($pb);

break;

}

print_r($row);

$rowCount++;

break;

case 2:

echo '[chat_server] => '.readString($pb)."\n";

break;

case 3:

echo '[chat_id] => '.readVarInt32($pb)."\n";

break;

case 4:

echo '[mission] => '.readVarInt32($pb)."\n";

break;

case 5:

echo '[max_limit] => '.readVarInt32($pb)."\n";

break;

case 6:

echo '[source] => '.readString($pb)."\n";

break;

case 7:

echo '[ds] => '.readVarInt32($pb)."\n";

break;

case 8:

echo '[de] => '.readVarInt32($pb)."\n";

break;

case 9:

echo '[max_count] => '.readVarInt32($pb)."\n";

break;

case 10:

echo '[realname] => '.readVarInt32($pb)."\n";

break;

case 11:

echo '[sectionlen] => '.readVarInt32($pb)."\n";

break;

default:

switch($type){

case 0:

echo '--Unknown ID '.$id.' with varint32 value '.readVarInt32($pb)."\n";

break;

case 1:

echo '--Unknown ID '.$id.' with fixed 64bit data hex '.bin2hex(fread($pb,8))."\n";

break;

case 5:

echo '--Unknown ID '.$id.' with fixed 32bit data hex '.bin2hex(fread($pb,4))."\n";

break;

default:

echo '--Unknown ID '.$id.' with type '.$type."\n";

}

echo $rowCount.' rows in file'."\n";

高潮来了

esterTion ( ͡° ͜ʖ ͡°)