2024-11-25 19:21:09,813 - Total .npz files found: 551
2024-11-25 19:21:09,814 - Training files: 440, Inference files: 111
2024-11-25 19:21:17,688 - Loading initial weights from /root/autodl-tmp/sam/sam_vit_h_4b8939.pth
2024-11-25 19:21:19,704 - Starting training...
2024-11-25 19:21:35,435 - Epoch 1/100, Loss: 2.2129
2024-11-25 19:21:53,289 - Epoch 2/100, Loss: 0.4174
2024-11-25 19:22:11,797 - Epoch 3/100, Loss: 0.2698
2024-11-25 19:22:30,193 - Epoch 4/100, Loss: 0.1088
2024-11-25 19:22:48,754 - Epoch 5/100, Loss: 0.0506
2024-11-25 19:23:07,242 - Epoch 6/100, Loss: 0.0367
2024-11-25 19:23:25,811 - Epoch 7/100, Loss: 0.0306
2024-11-25 19:23:44,566 - Epoch 8/100, Loss: 0.0282
2024-11-25 19:24:03,090 - Epoch 9/100, Loss: 0.0256
2024-11-25 19:24:21,479 - Epoch 10/100, Loss: 0.0242
2024-11-25 19:24:39,758 - Epoch 11/100, Loss: 0.0229
2024-11-25 19:24:57,962 - Epoch 12/100, Loss: 0.0221
2024-11-25 19:25:16,381 - Epoch 13/100, Loss: 0.0213
2024-11-25 19:25:34,972 - Epoch 14/100, Loss: 0.0208
2024-11-25 19:25:53,314 - Epoch 15/100, Loss: 0.0200
2024-11-25 19:26:11,815 - Epoch 16/100, Loss: 0.0196
2024-11-25 19:26:30,368 - Epoch 17/100, Loss: 0.0192
2024-11-25 19:26:48,843 - Epoch 18/100, Loss: 0.0190
2024-11-25 19:27:07,507 - Epoch 19/100, Loss: 0.0191
2024-11-25 19:27:22,410 - Epoch 20/100, Loss: 0.0187
2024-11-25 19:27:40,789 - Epoch 21/100, Loss: 0.0181
2024-11-25 19:27:59,294 - Epoch 22/100, Loss: 0.0178
2024-11-25 19:28:17,938 - Epoch 23/100, Loss: 0.0175
2024-11-25 19:28:36,305 - Epoch 24/100, Loss: 0.0172
2024-11-25 19:28:54,580 - Epoch 25/100, Loss: 0.0173
2024-11-25 19:29:09,385 - Epoch 26/100, Loss: 0.0171
2024-11-25 19:29:27,767 - Epoch 27/100, Loss: 0.0168
2024-11-25 19:29:46,126 - Epoch 28/100, Loss: 0.0165
2024-11-25 19:30:04,661 - Epoch 29/100, Loss: 0.0164
2024-11-25 19:30:23,119 - Epoch 30/100, Loss: 0.0162
2024-11-25 19:30:41,532 - Epoch 31/100, Loss: 0.0163
2024-11-25 19:30:56,377 - Epoch 32/100, Loss: 0.0163
2024-11-25 19:31:11,200 - Epoch 33/100, Loss: 0.0160
2024-11-25 19:31:29,458 - Epoch 34/100, Loss: 0.0159
2024-11-25 19:31:48,025 - Epoch 35/100, Loss: 0.0156
2024-11-25 19:32:06,576 - Epoch 36/100, Loss: 0.0154
2024-11-25 19:32:24,934 - Epoch 37/100, Loss: 0.0152
2024-11-25 19:32:43,172 - Epoch 38/100, Loss: 0.0153
2024-11-25 19:32:57,948 - Epoch 39/100, Loss: 0.0153
2024-11-25 19:33:12,723 - Epoch 40/100, Loss: 0.0154
2024-11-25 19:33:27,494 - Epoch 41/100, Loss: 0.0153
2024-11-25 19:33:42,317 - Epoch 42/100, Loss: 0.0150
2024-11-25 19:34:01,343 - Epoch 43/100, Loss: 0.0148
2024-11-25 19:34:20,971 - Epoch 44/100, Loss: 0.0148
2024-11-25 19:34:39,399 - Epoch 45/100, Loss: 0.0147
2024-11-25 19:34:57,957 - Epoch 46/100, Loss: 0.0149
2024-11-25 19:35:12,915 - Epoch 47/100, Loss: 0.0152
2024-11-25 19:35:27,747 - Epoch 48/100, Loss: 0.0152
2024-11-25 19:35:42,638 - Epoch 49/100, Loss: 0.0146
2024-11-25 19:36:00,952 - Epoch 50/100, Loss: 0.0149
2024-11-25 19:36:15,717 - Epoch 51/100, Loss: 0.0144
2024-11-25 19:36:33,461 - Epoch 52/100, Loss: 0.0143
2024-11-25 19:36:51,955 - Epoch 53/100, Loss: 0.0142
2024-11-25 19:37:10,305 - Epoch 54/100, Loss: 0.0142
2024-11-25 19:37:28,707 - Epoch 55/100, Loss: 0.0146
2024-11-25 19:37:43,500 - Epoch 56/100, Loss: 0.0142
2024-11-25 19:37:58,330 - Epoch 57/100, Loss: 0.0144
2024-11-25 19:38:13,141 - Epoch 58/100, Loss: 0.0141
2024-11-25 19:38:31,560 - Epoch 59/100, Loss: 0.0139
2024-11-25 19:38:49,814 - Epoch 60/100, Loss: 0.0140
2024-11-25 19:39:04,621 - Epoch 61/100, Loss: 0.0138
2024-11-25 19:39:23,102 - Epoch 62/100, Loss: 0.0136
2024-11-25 19:39:41,514 - Epoch 63/100, Loss: 0.0137
2024-11-25 19:39:56,340 - Epoch 64/100, Loss: 0.0135
2024-11-25 19:40:14,899 - Epoch 65/100, Loss: 0.0135
2024-11-25 19:40:29,847 - Epoch 66/100, Loss: 0.0137
2024-11-25 19:40:44,714 - Epoch 67/100, Loss: 0.0136
2024-11-25 19:40:59,586 - Epoch 68/100, Loss: 0.0134
2024-11-25 19:41:17,582 - Epoch 69/100, Loss: 0.0133
2024-11-25 19:41:36,105 - Epoch 70/100, Loss: 0.0133
2024-11-25 19:41:54,730 - Epoch 71/100, Loss: 0.0133
2024-11-25 19:42:09,715 - Epoch 72/100, Loss: 0.0134
2024-11-25 19:42:24,872 - Epoch 73/100, Loss: 0.0133
2024-11-25 19:42:39,888 - Epoch 74/100, Loss: 0.0131
2024-11-25 19:42:57,840 - Epoch 75/100, Loss: 0.0132
2024-11-25 19:43:12,831 - Epoch 76/100, Loss: 0.0130
2024-11-25 19:43:31,385 - Epoch 77/100, Loss: 0.0130
2024-11-25 19:43:49,767 - Epoch 78/100, Loss: 0.0130
2024-11-25 19:44:08,299 - Epoch 79/100, Loss: 0.0128
2024-11-25 19:44:26,694 - Epoch 80/100, Loss: 0.0128
2024-11-25 19:44:41,637 - Epoch 81/100, Loss: 0.0130
2024-11-25 19:44:56,594 - Epoch 82/100, Loss: 0.0129
2024-11-25 19:45:11,524 - Epoch 83/100, Loss: 0.0129
2024-11-25 19:45:26,475 - Epoch 84/100, Loss: 0.0128
2024-11-25 19:45:45,002 - Epoch 85/100, Loss: 0.0127
2024-11-25 19:46:03,564 - Epoch 86/100, Loss: 0.0128
2024-11-25 19:46:18,478 - Epoch 87/100, Loss: 0.0130
2024-11-25 19:46:33,344 - Epoch 88/100, Loss: 0.0126
2024-11-25 19:46:51,936 - Epoch 89/100, Loss: 0.0124
2024-11-25 19:47:10,156 - Epoch 90/100, Loss: 0.0130
2024-11-25 19:47:25,047 - Epoch 91/100, Loss: 0.0126
2024-11-25 19:47:39,978 - Epoch 92/100, Loss: 0.0126
2024-11-25 19:47:54,910 - Epoch 93/100, Loss: 0.0123
2024-11-25 19:48:12,837 - Epoch 94/100, Loss: 0.0124
2024-11-25 19:48:27,742 - Epoch 95/100, Loss: 0.0123
2024-11-25 19:48:46,265 - Epoch 96/100, Loss: 0.0122
2024-11-25 19:49:04,786 - Epoch 97/100, Loss: 0.0121
2024-11-25 19:49:23,395 - Epoch 98/100, Loss: 0.0120
2024-11-25 19:49:41,836 - Epoch 99/100, Loss: 0.0120
2024-11-25 19:50:00,327 - Epoch 100/100, Loss: 0.0122
2024-11-25 19:50:00,327 - Training completed.
2024-11-25 19:50:00,327 - Starting inference...
2024-11-25 19:50:00,413 - Sample 1: Step Time: 0.0539s, Memory Used: 7732.31 MB, Power Usage: 79.56 W
2024-11-25 19:50:00,454 - Sample 2: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 79.70 W
2024-11-25 19:50:00,493 - Sample 3: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 79.70 W
2024-11-25 19:50:00,531 - Sample 4: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 79.95 W
2024-11-25 19:50:00,570 - Sample 5: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 79.95 W
2024-11-25 19:50:00,608 - Sample 6: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 79.95 W
2024-11-25 19:50:00,645 - Sample 7: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 68.55 W
2024-11-25 19:50:00,683 - Sample 8: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 68.55 W
2024-11-25 19:50:00,722 - Sample 9: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 68.87 W
2024-11-25 19:50:00,759 - Sample 10: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 68.87 W
2024-11-25 19:50:00,797 - Sample 11: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 68.87 W
2024-11-25 19:50:00,835 - Sample 12: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 69.11 W
2024-11-25 19:50:00,875 - Sample 13: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 69.11 W
2024-11-25 19:50:00,912 - Sample 14: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 69.11 W
2024-11-25 19:50:00,950 - Sample 15: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 69.48 W
2024-11-25 19:50:00,987 - Sample 16: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 69.48 W
2024-11-25 19:50:01,026 - Sample 17: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 69.74 W
2024-11-25 19:50:01,063 - Sample 18: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 69.74 W
2024-11-25 19:50:01,100 - Sample 19: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 69.74 W
2024-11-25 19:50:01,137 - Sample 20: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 70.15 W
2024-11-25 19:50:01,174 - Sample 21: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 70.15 W
2024-11-25 19:50:01,211 - Sample 22: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.15 W
2024-11-25 19:50:01,249 - Sample 23: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.53 W
2024-11-25 19:50:01,287 - Sample 24: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.53 W
2024-11-25 19:50:01,324 - Sample 25: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.84 W
2024-11-25 19:50:01,361 - Sample 26: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.84 W
2024-11-25 19:50:01,399 - Sample 27: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 70.84 W
2024-11-25 19:50:01,436 - Sample 28: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 62.48 W
2024-11-25 19:50:01,473 - Sample 29: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 62.48 W
2024-11-25 19:50:01,511 - Sample 30: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 62.48 W
2024-11-25 19:50:01,548 - Sample 31: Step Time: 0.0059s, Memory Used: 7732.31 MB, Power Usage: 62.71 W
2024-11-25 19:50:01,585 - Sample 32: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 62.71 W
2024-11-25 19:50:01,623 - Sample 33: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 62.74 W
2024-11-25 19:50:01,660 - Sample 34: Step Time: 0.0060s, Memory Used: 7732.31 MB, Power Usage: 62.74 W
2024-11-25 19:50:01,697 - Sample 35: Step Time: 0.0059s, Memory Used: 7732.31 MB, Power Usage: 62.74 W
2024-11-25 19:50:01,734 - Sample 36: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:01,772 - Sample 37: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:01,810 - Sample 38: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:01,848 - Sample 39: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.67 W
2024-11-25 19:50:01,885 - Sample 40: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.67 W
2024-11-25 19:50:01,924 - Sample 41: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.70 W
2024-11-25 19:50:01,961 - Sample 42: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 62.70 W
2024-11-25 19:50:01,999 - Sample 43: Step Time: 0.0061s, Memory Used: 7732.31 MB, Power Usage: 62.70 W
2024-11-25 19:50:02,036 - Sample 44: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.73 W
2024-11-25 19:50:02,074 - Sample 45: Step Time: 0.0062s, Memory Used: 7732.31 MB, Power Usage: 62.73 W
2024-11-25 19:50:02,112 - Sample 46: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.73 W
2024-11-25 19:50:02,150 - Sample 47: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.90 W
2024-11-25 19:50:02,188 - Sample 48: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.90 W
2024-11-25 19:50:02,225 - Sample 49: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.79 W
2024-11-25 19:50:02,263 - Sample 50: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.79 W
2024-11-25 19:50:02,301 - Sample 51: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.79 W
2024-11-25 19:50:02,339 - Sample 52: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.81 W
2024-11-25 19:50:02,378 - Sample 53: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.81 W
2024-11-25 19:50:02,416 - Sample 54: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.81 W
2024-11-25 19:50:02,453 - Sample 55: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.92 W
2024-11-25 19:50:02,491 - Sample 56: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.92 W
2024-11-25 19:50:02,530 - Sample 57: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.80 W
2024-11-25 19:50:02,568 - Sample 58: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.80 W
2024-11-25 19:50:02,606 - Sample 59: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.80 W
2024-11-25 19:50:02,671 - Sample 60: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.77 W
2024-11-25 19:50:02,716 - Sample 61: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.77 W
2024-11-25 19:50:02,756 - Sample 62: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.73 W
2024-11-25 19:50:02,794 - Sample 63: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.73 W
2024-11-25 19:50:02,832 - Sample 64: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:02,870 - Sample 65: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:02,908 - Sample 66: Step Time: 0.0067s, Memory Used: 7732.31 MB, Power Usage: 62.64 W
2024-11-25 19:50:02,946 - Sample 67: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:02,984 - Sample 68: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:03,022 - Sample 69: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:03,060 - Sample 70: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:03,098 - Sample 71: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:03,136 - Sample 72: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.50 W
2024-11-25 19:50:03,175 - Sample 73: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.50 W
2024-11-25 19:50:03,213 - Sample 74: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.50 W
2024-11-25 19:50:03,251 - Sample 75: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.48 W
2024-11-25 19:50:03,289 - Sample 76: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.48 W
2024-11-25 19:50:03,327 - Sample 77: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.47 W
2024-11-25 19:50:03,365 - Sample 78: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.47 W
2024-11-25 19:50:03,402 - Sample 79: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.47 W
2024-11-25 19:50:03,440 - Sample 80: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.44 W
2024-11-25 19:50:03,478 - Sample 81: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.44 W
2024-11-25 19:50:03,516 - Sample 82: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.44 W
2024-11-25 19:50:03,553 - Sample 83: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.42 W
2024-11-25 19:50:03,591 - Sample 84: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.42 W
2024-11-25 19:50:03,629 - Sample 85: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.42 W
2024-11-25 19:50:03,666 - Sample 86: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.42 W
2024-11-25 19:50:03,704 - Sample 87: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.42 W
2024-11-25 19:50:03,742 - Sample 88: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.39 W
2024-11-25 19:50:03,780 - Sample 89: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.39 W
2024-11-25 19:50:03,818 - Sample 90: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.39 W
2024-11-25 19:50:03,856 - Sample 91: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.47 W
2024-11-25 19:50:03,893 - Sample 92: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.47 W
2024-11-25 19:50:03,932 - Sample 93: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:03,970 - Sample 94: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:04,008 - Sample 95: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:04,047 - Sample 96: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.61 W
2024-11-25 19:50:04,084 - Sample 97: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.61 W
2024-11-25 19:50:04,123 - Sample 98: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.58 W
2024-11-25 19:50:04,162 - Sample 99: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.58 W
2024-11-25 19:50:04,200 - Sample 100: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.58 W
2024-11-25 19:50:04,238 - Sample 101: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.60 W
2024-11-25 19:50:04,276 - Sample 102: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.60 W
2024-11-25 19:50:04,314 - Sample 103: Step Time: 0.0069s, Memory Used: 7732.31 MB, Power Usage: 62.60 W
2024-11-25 19:50:04,352 - Sample 104: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:04,391 - Sample 105: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.54 W
2024-11-25 19:50:04,429 - Sample 106: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:04,467 - Sample 107: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:04,506 - Sample 108: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.55 W
2024-11-25 19:50:04,545 - Sample 109: Step Time: 0.0065s, Memory Used: 7732.31 MB, Power Usage: 62.52 W
2024-11-25 19:50:04,583 - Sample 110: Step Time: 0.0063s, Memory Used: 7732.31 MB, Power Usage: 62.52 W
2024-11-25 19:50:04,621 - Sample 111: Step Time: 0.0064s, Memory Used: 7732.31 MB, Power Usage: 62.45 W
2024-11-25 19:50:04,621 - Inference completed.
2024-11-25 19:50:04,622 - Average First Token Latency: 0.0520s
2024-11-25 19:50:04,622 - Average Token Latency: 0.0051s
2024-11-25 19:50:04,622 - TP50 Token Latency: 0.0047s
2024-11-25 19:50:04,622 - TP99 Token Latency: 0.0052s
2024-11-25 19:50:04,622 - Average Throughput per Token: 13827657.5310 tokens/s
2024-11-25 19:50:04,623 - Average Step Time: 0.0067s
2024-11-25 19:50:04,623 - Average Memory Usage: 7732.31 MB
2024-11-25 19:50:04,623 - Average Power Usage: 64.88 W
2024-11-25 19:50:04,706 - Inference results and GPU metrics saved.